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(57) Abstract 

Color image enhancement apparatus (10) comprising apparatus (12) for receiving signals representing a color image, im- 
age processing apparatus, employing the received signals, for image processing of the high spatial frequency chromatic compo- 
nents of a color image, and apparatus for providing a color image from the output of said image processing apparatus. 
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APPARATUS AND METHOD FOR ENHANCING COLOR IMAGES 
FIELD OF THE INVENTION 

The present invention relates to image processing 
generally and more particularly to detail enhancement in color 
images . 

BACKGROUND OP THE INVENTION 

Color image processing is discussed generally in the 
following texts, the disclosures of which are hereby incorporated 
by reference: 

Rosenfeld, A, and Kak, A. C. Digital picture 
processinQ . Academic Press, 1982, and 

Wysecki, G. and Stiles, S., Color science: gp^ggptg 
and methods, cmantitiative data and formulae. John Wiley a-id 
Sons, 1982. 
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SUMMARY OF THE INVENTION 

The present invention seeks to provide an improved 
system for enhancing color images which is based on an adaptive"^ 
opponent color model. 

The present invention also seeks to provide a system 
for enhancing color images which takes into account properties of 
the human visual system (HVS) . Models of the human vision system 
are described in Chapters 1 and 2 of Appendix A, appended hereto. 

The present invention also seeks to provide a system 
for adapting a color image for perception j by color blind 
individuals. 

There is thus provided, in accordance with a preferred 
embodiment of the present invention, color image enhancement 
apparatus including apparatus for receiving signals representing 
a color image, image processing apparatus, employing the received 
signals, for image processing of the high spatial frequency 
chromatic components of the color image, and apparatus for 
providing a color image using the output of the image processing 
apparatus . 

Further in accordance with a preferred embodiment of 
the present invention, the apparatus for providing a color image 
also employs an achromatic component of the color image which was 
not processed by the image processing apparatus. 

Still further in accordance with a preferred embodiment - 
of the present invention, the apparatus for providing a color 
image also employs an achromatic component of the color image 
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which was processed by the Image processing apparatus. 

Still further in accordance with a preferred embodiment 
of the present invention, the apparatus for providing a color 
image also employs an achromatic component of the color image. 

Additionally in accordance with a preferred embodiment 
of the present invention, the apparatus for receiving includes 
apparatus for receiving signals representing a color image in a 
first color space, and apparatus for transforming the received 
signals from the first color space into a color space which 
simulates the cones of the human visual system. 

Further in accordance with a preferred embodiment of 
the present invention, the first color space includes an RGB 
space and the cone-simulating color space includes a (Vj^, Vj^, Vg) 
color space. 

Additionally in accordance with a preferred embodiment 
of the present invention, the apparatus for providing includes 
apparatus for transforming the output of the image processing 
apparatus from the cone-simulating color space to a second color 
space . 

Still further in accordance with a preferred embodiment 
of the present invention, the second color space includes an RGB 
space • 

Additionally in accordance with a preferred embodiment 
of the present invention, the apparatus for receiving includes 
apparatus for at least partially decorrelating signals 
representing the color image. 

Further in accordance with a preferred embodiment of 
the present invention, the apparatus for decorrelating includes 
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apparatus for performing a Karhunen-Loeve transform. 

Still further in accordance with a preferred embodiment 
of the present invention, the apparatus for providing includes 
apparatus for performing an inverse of the Karhunen-Loeve ^ 
transform. 

There is also provided, in accordance with another 
preferred embodiment of the present invention, color image 
enhancement apparatus including apparatus for image processing of 
the high spatial frequency chromatic components of a color image. 

There is further provided, in accordance with another 
preferred embodiment of the present invention, color image 
enhancement apparatus including apparatus for image processing of 
the high spatial frequency chromatic components of a color image, 
whereby a resulting enhanced color image is produced at least 
mainly from the high spatial frequency components. 

Further in accordance with a preferred embodiment of 
the present invention, the high spatial frequency components are 
those contained in the top half, top third, or top quarter of the 
spatial frequency range of the color image. 

Further in accordance with a preferred embodiment of 
the present invention, the power spectrum of the high spatial 
frequency chrom.atic components is similar to that of a high 
spatial frequency achromatic component of the color image* 

There is also provided, in accordance with yet a 
further preferred embodiment of the present invention, color 
image enhancement apparatus for modifying a color image for 
perception by a color-blind individual, the apparatus including 
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apparatus for receiving signals representing the color image, 
image processing apparatus including apparatus^ employing the 
received signals, for modifying the color image such that at 
least one color in the color image which a color-blind individual 
cannot differentiate is transformed to at least one color which 
the color-blind individual can differentiate, and apparatus for 
providing a color image using the output of the image processing 
apparatus « 

Further in accordance with a preferred embodiment of 
the present invention, apparatus for identifying at least one 
color which the color-blind individual does not differentiate is 
also provided. 

Further in accordance with a preferred embodiment of 
the present invention, the apparatus for modifying includes 
apparatus, employing the received signals, for modifying the 
color image such that a color in the color image which a 
dichromate does not differentiate is transformed to at least one 
color which the dichromate can differentiate. 

Still further in accordance with a preferred embodiment 
of the present invention, the apparatus for modifying includes 
apparatus, employing the received signals, for modifying the 
color image such that at least one color in the color image which 
a raonochromate does not differentiate is transformed to a color 
which the monochromate can differentiate* 

Additionally in accordance with a preferred embodiment 
of the present invention, the image processing apparatus includes 
apparatus for enhancing chromatic differences between spatially 
adjacent colors. 
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Further in accordance with a preferred embodiment of 
the present invention^ the apparatus for enhancing chromatic 
differences includes apparatus for enhancing chromatic 
differences between spatially adjacent colors which are , 
achromatically indiff erentiable. 

* 

Still further in accordance with a preferred embodiment 
of the present invention, the apparatus for enhancing chromatic 
differences includes apparatus for enhancing chromatic 
differences between spatially adjacent colors which are 
achromatically diff erentiable. 

Additionally in accordance with a preferred embodiment 
of the present invention, apparatus is also provided for 
enhancing primarily a high spatial frequency portion of an 
achromatic component of the color image. 

Further in accordance with a preferred embodiment of 
the present invention, apparatus is also provided for enhancing 
in the presence of noise including apparatus for enhancing 
primarily a low spatial frequency portion oi an achromatic 
component of the color image. 

There is also provided, in accordance with another 
preferred embodiment of the present invention, a method for color 
image enhancement including the steps of receiving signals 
representing a color image, employing the received signals for 
image processing of the high spatial frequency chromatic 
components of the color image, and providing a color image using 
the output of the apparatus for image processing. 

Further in accordance with a preferred embodiment of 
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the present invention, the step of providing a color image also 
employs an achromatic component of the color image which was not 
p^o<^-essed in the image processing step. 

Still further in accordance with a preferred embodiment 
of the present invention, the step of providing a color image 
also employs an achromatic component of the color image which was 
processed in the image processing step. 

Additionally in accordance with a preferred embodiment 
of the present invention, the step of providing a color image 
also employs an achromatic component of the color image. 

Further in accordance with a preferred embodiment of 
the present invention, the step of receiving includes the steps 
of receiving signals representing a color image in a first color 
space, and transforming the received signals from the first color 
space into a color space which simulates the cones of the human 
vi f^Mal system. 

Still further in accordance with a preferred embodiment 
of the present invention, the first color space ^includes an RGB 
space and the cone-simulating color space includes a (V^/ V^, Vg) 
color space. 

Further in accordance with a preferred embodiment of 
the present invention, the step of providing includes the step of 
transforming the output of the employing step from the cone- 
simulating color space to a second color space. 

Still further in accordance with a preferred embodiment 
of the present invention, the second color space includes an RGB 
space. 

Additionally in accordance with a preferred embodiment 
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of the present invention, the step of receiving includes the step 
of at it^cio!;. nartially decorrelating signals representing the 
color image. 

Further in accordance with a preferred embodiment of * 
the present invention, the step of decorrelating includes the 
step of performing a Karhunen-Loeve transform. 

Still further in accordance with a preferred embodiment 
of the present invention, the step of providing includes the step 
of performing an inverse of the Karhunen-Loeve transform. 

There is also provided, in accordance with another 
preferred embodiment of the present invention, a color image 
enhancement method including the step of image processing of the 
high spatial frequency chromatic components of a color image. 

There is further provided, in accordance with another 
preferred embodiment of the present invention, a color image 
eni^^ncement method including the step of image processing of the 
high spatial frequency chromatic components of a color image, 
whereby a resulting enhanced color image is produced at least 
mainly from the high spatial frequency components. 

Further in accordance with a preferred embodiment of 
the present invention, the high spatial frequency components 
include the top half, top third or top quarter of the spatial 
frequency range of the color image. 

Still further in accordance with a preferred embodiment 
of the present invention, the power spectrum of the high spatial 
frequency chromatic components is similar to that of a high 
spatial frequency achromatic component of the color image. 
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Additionally in accordance with a preferred erabodiraent 
of the present invention, there is provided a color image 
enhancement method for modifying a color image for perception by 
a color-blind individual, the method including the steps of 
receiving signals representing the color image, image processing 
the color image, including the step of employing the received 
signals for modifying the color image such that at least one 
color in the color image which a color-blind individual cannot 
differentiate is transformed to at least one color which a color- 
blind individual can differentiate, and providing a color image 
using the output of the apparatus for image processing. 

Further in accordance with a preferred embodiment of 
the present invention, the method also includes the step of 
identifying at least one color which the color-blind individual 
does not differentiate. 

Still further in accordance with a preferred embodiment 
of the present invention, the step of employing includes the step 
of employing the received signals for modifying .the color image 
such that a color in the color image which a dichromate does not 
differentiate is transformed to at least one color which the 
dichromate can differentiate. 

Additionally in accordance with a preferred embodiment 
of the present invention, the step of employing includes the step 
of employing the received signals for modifying the color image 
such that at least one color in the color image which a 
monochroraate does not differentiate is transformed to a color 
which the monochromate can differentiate. 

Further in accordance with a preferred embodiment of 
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the present invention, the step of image processing includes the 
step of enhancing chromatic differences between spatially 
adjacent colors. 

Still further in accordance with a preferred embodiment * 
of the present invention, the step of enhancing chromatic 
differences includes the step of enhancing chromatic differences 
between spatially adjacent colors which are achroraatically 
indif f erentiable . 

Still further in accordance with a preferred embodiment 
of the present invention, the step of enhancing chromatic 
differences includes the step of enhancing chromatic differences 
between spatially adjacent colors which are achromatically 
diff erentiable • 

Additionally in accordance with a preferred embodiment 
of the present invention, the method includes the step of 
enhancing primarily a high spatial frequency portion of an 
achromatic component of the color image. 

Still further in accordance with a preferred embodiment 
of the present invention, the method includes a step of enhancing 
in the presence of noise including the step of enhancing 
primarily a low spatial frequency portion of an achromatic 
component of the color image. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood and 
appreciated from the following detailed description, taken in 
conjunction with the drawings in which: 

Fig. 1 is a simplified block diagram of image 
enhancement apparatus constructed and operative in accordance 
with a first preferred embodiment of the present invention; and 

Fig. 2 is a simplified block diagram of image 
enhancement apparatus constructed and operative in accordance 
with a second preferred embodiment of the present invention. 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Reference is now made to Fig. 1 which illustrates image 
enhancement apparatus 10 constructed and operative in accordance 
with a first preferred embodiment of the present invention. 

Image enhancement apparatus 10 comprises an input 
space/cone space converter 12, a decorrelator 20, an image 
enhancement unit 40, a chromatic/achromatic recorrelator 50 and a 
cone space/output space converter 60. 

The image enhancement apparatus 10 receives signals 
representing a color image in a color space such as RGB or YC^Cj^. 
The source of these signals may, for example, be a color video 
CCD camera 70. 

Preferably, the input signals are converted by input 
space/cone space converter 12 to a representation similar to that 
employed by HVS cones, which are biological retinal receptors. 
One example of a cone-simulating color space is a (Vj^, Vj^, Vg) 
space, as described in Appendix A. However, any other suitable 
cone-simulating color space may be employed. 

The output of input space/cone space converter 12 is 
provided to decorrelator 20 which is operative to reduce the 
correlations between the plurality of components of the output of 
converter 12. 

According to a preferred embodiment of the present 
invention, decorrelator 20 is based upon the Karhunen-Loeve 
transform (KLT) , which is described in the following publication, 
the disclosure of which is incorporated herein by reference: 

Gonzalez, R. C. and Wintz, P. Digital image 
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processino . Addison-Wesley, 1987, pp. 122-130. 

Decorrelator 20 preferably comprises a decorrelation 
transform computation unit 80. and a decorrelation transformer 82. 
Decorrelation transform computation unit 80 computes the 
transformation to be applied to the output of converter 12, as 
described in more detail on page 8 of the document appended 
hereto and referenced Appendix B. Decorrelation transformer 82 
applies the transformation to the converter 12 output. 

Alternatively, decorrelation transform computation unit 
80 may be eliminated. Instead, a KLT transform may be precomputed 
based on a sample of images which is preferably representative of 
the type of images for which it is desired to employ the 
apparatus of Fig. 1. For example, as explained in more detail in 
Appendix B, a KLT transform suitable for daylight landscapes may 
be computed based upon a suitable sample including a variety of 
daylight landscapes. 

In the illustrated embodiment, the components of the 
output of decorrelator 20, preferably comprising ^cone-simulating, 
decorrelated image data, are labelled black/white, red/green and 
blue/yellow, respectively, to facilitate an intuitive 
and rapid understanding of the present invention. However, it is 
appreciated that the output channels of decorrelator 20 may not, 
in fact, correspond exactly to this verbal description. 

The cone-simulating, decorrelated image data, also 
termed herein (Kj^, Kj, K3), is provided to image enhancement 
unit 40. A particular feature of the present invention is that 
image enhancement unit 40 preferably processes high spatial 
frequency chromatic components of the color image data and that 

13 



SUBSTITUTE SHEET (RULE 26) 



wo 94/1 1987 



PCr/US93/11146 



the color Image data output by image enhancement unit 40 is 
preferably produced mainly or entirely by processing of the high 
spatial frequency chromatic components. 

In the present specification, the term "high spatial 
frequency chromatic components" refers to chromatic components 
whose spatial frequency is included in the top half, third or 
quarter, on a log scale, of the spatial frequency range of the 
entire color image. For example, if the size of the color image 
is 512 pixels x 512 pixels, the maximum spatial frequency along a 
particular dimension of the color image is 512/2 - 256 cycles per 
image, and any of the following categories of chromatic 
components may be regarded as "high spatial freqniency chromatic 
components " : 

"top half" chromatic components whose spatial frequency 
is greater than 256^/^ = 16 cycles per image; 

"top third" chromatic components whose spatial 
frequency is greater than 256^/^ = approximately 40 cycles per 
image; and ^ 

"top quarter" chromatic components whose spatial 
frequency is greater than 256^/^ = 64 cycles per image. 

Alternatively, the term "high spatial frequency 
chromatic components" refers to those chromatic components having 
a power spectrum resembling the power spectrum of high spatial 
frequency achromatic components of the color image. The term 
"high spatial frequency achromatic components" may be defined 

4 

analogously to the previous definitions in which "high spatial 
frequency chromatic components" were defined independent of - 
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achromatic components . 

An operational definition of the term "high spatial 
frequency chromatic components" in applications in which high- 
pass achromatic enhancement is provided, is those chromatic 
components which survive filtering with the same high-pass 
filters employed for the achromatic enhancement. 

Preferably, color enhancement unit 40 is operative to 
enhance details in the color image by enhancing chromatic 
differences between spatially adjacent colors which may or may 
not be achromatically dif f erentiable. 

Preferably, the color enhancement unit 40 is also 
operative to process achromatic components of the color image. 
According to one alternative embodiment of the present invention, 
primarily the high frequency portion of the achromatic components 
is processed. 

Alternatively, the achromatic processing performed by 
color enhancement unit 40 primarily comprises processing of the 
low frequency portion of the achromatic components of the color 
image. This embodiment is particularly useful in the presence of 
strong noise, because empirical findings indicate that most of 
the noise occurs in the achromatic components. 

Alternatively, in the presence of noise, even the low 
pass filtering of the achromatic components may be eliminated and 
processing may be restricted only to the chromatic components of 
the color image. 

The output of color image enhancement unit 40 is 
provided to recorrelating unit 50 which performs the inverse of 
the transformation applied by decorrelating unit 82. The inverse 
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transform is computed by inverse transform computation unit 90 
which receives an indication of the original transform from 
decorrelating transform computation unit 80. 

Alternatively, if decorrelating transform computation . 
unit 80 is eliminated, as explained above, recorrelation 
transform computation unit 90 may be eliminated and the inverse 
transform employed by recorrelating unit 50 may be predetermined, 
using a representative set of color images, as explained above. 

The output of recorrelating unit 50 is provided to cone 
space/output space converter 60 which is operative to convert the 
data from cone space representation to output space 
representation. Any suitable output space may be employed, such 
as the RGB space of a CRT. The output space may or may not be the 
same as the input space in which the data received by chromatic 
enhancement apparatus 10 is represented. 

The output of the apparatus 10 is an enhanced color 
image, on any suitable medium such as a CRT 94. 

A detailed description of computational features of the 
various components of the apparatus of Fig. 1 appears in Appendix 
B. Other features of the apparatus of Fig. 1 are described in 
Appendix A. 

It is appreciated that, in actual practice, many 
modifications are possible. For example, of course, instead of 
providing a sequence of units for performing a sequence of 
transformations on input color image data, a single unit may be 
provided which performs a single transformation which is the 
composition of the sequence of transformations. In other words, 
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the result of applying the single transformation is the same as 
the result of sequentially applying the sequence of 
transformations. 

Reference is now made to Fig. 2 which illustrates a 
system for adapting color images for perception by color blind 
individuals, and which is based on the apparatus described above 
with reference to Pig. 1. Substantially identical elements in 
Figs. 1 and 2 are identically numbered for easy reference and 
are not described again hereinbelow for brevity. 

In Fig. 2, a color image, processing unit 100 is 
operative to "cross-enhance" at least one chromatic channel of 
the color image which is not perceived by a color blind 
individual using at least one other channel of the color image 
which is perceived by the color blind individual. For example^ in 
the case of an individual who does not differentiate red/green, 
edges in the red/green channel will be enhanced in either or 
both of the blue/yellow and black/white channels. 

A preferred method of operation for color image 
processing unit 100 is as follows: 

a. Diagnose the type of color blindness of a target 

individual, preferably employing a computerized color blindness 
diagnosis system 110 interacting with the color blind individual. 
The computerized color blindness diagnosis system 110 is 
operative to determine which chromatic channels are not 
perceived and may be based on any suitable conventional testing 
materials for diagnosing color blindness. Color blindness 
diagnosis system 110 may, for example, be based upon the 
the following publication and the publications cited thereby, the 
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disclosures of which are hereby incorporated by reference: 

Rushton, W. A. H., "Visual pigments and color 
blindness". Scientific American, Vol. 232(3), March 1975, pp. 64 
- 74. 

Alternatively, the color blindess diagnosis information 
may be provided manually or from a memory in which color 
blindness diagnoses were prestored. 

According to still a further alternative, the operation 
of image processing unit 100 may be customized to adapt color 
images for perception only by a predetermined type of color blind 
individual, such as a red/green color blind individual. 

In the illustrated embodiment, the image processing 
unit 100 is illustrated as "cross-enhancing" the red/green 
channel by "transferring" red/green information to other 
channels, namely the blue/yellow channel and/or the achromatic 
channel. It is appreciated, however, that image processing unit 
100 may be operative to "cross-enhance" any one or any subset of 
the channels received thereby. 

b. A gradient computation unit 120 is employed to detect 
gradients in the red/green channel. 

c. The detected gradients are transferred to the 
blue/yellow and/or achromatic channels by a gradient application 
unit 130, For example, the gradient or spatial derivative 
detected for the red/green channel is preferably multiplied by a 
calibrating constant and may then be added to either or both of 
the blue/yellow and black/white channels. 

It will be appreciated by persons skilled in the art 
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particularly shown and described hereinabove. Rather, the scope 
of the present invention is defined only by the claims that 
follow: 



19 

SUBSTITUTE SHEET (RULE 26) 



wo 94/1 1987 



PCr/US93/11146 



Contents 



List of Figures 
List of fCables 

Abstract ^ 
List of Symbols and Abbreviations 

5 

Introduction 

9 

Background to Colour Vision 

2.1 Trichromacy 

2.2 Colour Opponencv 

2.3 Opponent Colour Transform 

2.4 Contrast Sensitivitv 

Development of Colour Image Enhancement Computationaf Model 14 

3.1 Transformation of RGB to Cone Prunaries 

3.2 Separation of Chromatic and Acbromaticlaformation 1^ 

3.2.1 Discrete Karhunen-Loeve Transform 

3.2.2 Multispectral Discrete Karhunen-Loeve Transform 17 

3.3 The Enhancement Model and Its Relation to Other Models 19 

2X 

3.4 Scene Independent Transform 

3.5 Frequency Properties of Karhunen-Loeve Channels ^3 

2T 

3.5 Perceptual Interpretation of K Space 

30 

Use of Model in Colour Enhancement 

30 

4.1 Attempted Enhancement Methods • 

4.1.1 Amplitude Scaling 

31 

4.1.2 Histogram Methods 

4.2 Spatial Chromatic Enhancement 



20 



SUBSTITUTE SHEET (RULE 26) 



wo 94/11987 



PCT/US93/11146 



4.2.1 The Role of Retinal Lateral Inhibition . . . . 

4.2.2 Enhancement Based on Lateral Inhibition . . 

4.3 An Heuristic Measure of Colour Image Improvement 

4.4 Enhancement of Model Predicted Channels 

4,4*1 Achromatic Processing 

4.4.2 Chromatic Processing • 

4.5 Enhancement Using a Constant transform 

4.6 Enhancement in the Presence of Noise 

5 Summary and Conclusions 

6 References 

A Colourinaetry 

B Scene Independent Transform 



21 




wo 94/1 1987 



PCr/US93/11146 



List of Figures 



8 



1,1 Block diagram of Faugeras image processing model .• • • 

2.1 Vos-Walraven cone spectral sensitivity functions " 1° 

2.2 Psyciophysicallmninance and chromatic contrast sensitivity functions .... 13 

3.1 Block diagram of the generalized computational model ' 19 

3!2 Spectral sensitivity cnrves of the achromatic (JTi) and two chromatic channels, 

red-green (JTj) and blue-yellow (Ks) 

3.3 Block diagram of the simplified computational model 25 

3.4 Average logarithmic power spectral density of the Ki and K2 planes 2T 

3.5 Average ampUtude spectrum of the Ki, and K3 planes 28 

3.6 Perceptual quantities of Hue, Intensity and Saturation shown in K space ... 29 

4.1 Typical opponent channel histogram shown prior to and after peak shifting . 33 

4.2 Zero crossing step response of Laplacian filter and resulting Mach bands ... 36 

4.3 Block diagram of the enhancement used on each of the three chajmels .... 37 
4*4 Original 512 X 512 image of the "view". Colour resolution is 24 bits/pixel. . . 42 

4.5 Achromatic (Ki) enhancement of 'Sriew" image 43 

4.5 Chromatic enhancement of only the JTz channel of "view" image 45 

4.7 Chromatic enhancement of only the JTs channel of "view" image 46 

4 8 Chromatic enhancement of both and K3 channels of "view" image 47 

4.9 Both Achromatic (JTa) and Chromatic {K, and K») enhancement on "view" 

48 

image 

4.10 Separate achromatic, chromatic and combined achromatic-chromatic enhance- 
ment for "ship" im^e 

4 11 Separate achromatic, chromatic and combined achromatic-chromatic enhance- 

. . 50 

ment for "girl" image 

4.12 Achiomatic-chromaric enhancement on «view" image using both a constant 

transform and the full computational model 

4 13 Effect of separate Chromatic and Achromatic edge enhancement on noisy 

53 

"ship" miage 



22 



SUBSTRUTE SHEET (RULE 26) 



wo 94/1 1987 PCT/US93/1 1 146 

List of Tables 

3.1 Table, of signal energies in the various colour planes 25 

4.1 Valnes of the quality iadicators in eadi of the enhancement experiments. ... 51 



23 



SUBSTITUTE SHEET (RULE 26) 



wo 94/11987 



PCr/US93/11146 



Abstract 



We investigate an opponent colour model of the HVS (hnman visual system) and show how 
this model can be used for colour image enhancement, in particular, colour edge enhancement. 

The principle of the model is that in the HVS cone tristimtili are transformed to three 
opponent channels (black-white, red-green and blue-yellow) so that redundant information 
is minimized and energy compressed. Therefore after transforming a digital colour image 
represented by standard R, G and B tristimuli to cone tristimuli we use a Karhunen-Loeve 
transform to -simulate the HVS opponent transform. 

The model produces one all positive channel that contains at least 98 % of the total image 
energy. The spectral sensitivity function of this channel is very similar to the CIE luminous 
efficiency channel and thus contains the achromatic information. The model also results in 
two other antagonistic chamiels whose spectral sensitivity functions are similar to the Red- 
Green and Blue- Yellow opponent chromatic channels present in the HVS. The achromatic 
channel contains all spatial frequencies from dc upwards while the two chromatic channels 
are essentially lowpass. 

For a set of similar images the model may be simplified from the- general case of a 
Karhunen-Loeve transform per image to a simple linear matrix transformation. This is shown 
for a set as general as the set of natural daylight images. 

Based on recent evidence of opponent centre-surround red-green and blue-yellow retinal 
receptive fields we enhance chromatic information by creating colour Mach bands. This is 
done by convolving the chromatic channels with a digital Laplacian mask. The resultant zero 
crossings are then scaled by a parameter, a, and added to the original channel. 

The result is a sharper image with small colour detail highlighted and colours at least as 
rich as in the original image. Additional detail that achromatic edge enhancement does not 
affect is improved. This result is contrary to popular belief that since the chromatic compo- 
nents contain so little information, they need not be enhanced and can in fact be blurred. 
Because of limitations of the HVS very high frequency enhancement cannot be obtained using 
chromatic edge enhancement alone and the best result is obtained when combining standard 
achromatic enhancement with the novel chromatic edge enhancement. Recommended values 
of the parameter a are a = lin the achromatic channel and a = 3 in the chromatic channels. 

Because of the spatial frequency properties of the model channels, most high frequency 
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image noise is in the achromatic channel. Edge enhancement on the chromatic chaimels does 
not perceptibly increase the noise and thus can be used without fear of degrading the image. 

The results show that chromatic edge enhancement provides enhancement not possible 
using achromatic edge enhancement alone and that it may be used without degrading- the 
image even in the presence of high frequency noise. In addition, the simplified model and 
enhancement procedure proposed are both suitable for real time imaging systems. 
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Chapter 1 

Introduction 



The research presented in this report is concerned with the enhancement of digital colour 
images and a» such is also concerned with investigating the appropriate enhancement envi- 
ronment. 

When speaking of colour image enhancement it is necessary to make the distinction be- 
tween enhanc^ent for image imderstanding purposes, for example sateUite images, and for 
visual quaHty improvement such as television unages. In the first instance actual colours are 
insignificant and may be altered at will to make certain features more prominent. In the 
second type of enhancement, with which we axe concerned, true colours must be maintamed. 

There are twomain ways in which a colour image can be visuaUy enhanced. The first way is 
to alter the image such that undesirable colour casts are removed and the scene is rendered as 
if illuminated by a neutral white illuminant. In the Uterature this is referred to as the problem 
of colour constancy. The other main branch of colour image enhancement is enhancement of 
detaa, noise removal, debluiiing as well as other well known unage enhancement techniques. 

Although we are not concerned with colour constancy in this study we will discuss it briefly 
as it does introduce some of the techniques of dealing with colour images. The problem of 
colour constancy is related to how the HVS (human visual system) is consistently able to see 
the same colours in a scene even under different conditions of illumination whereas a camera 
Tised under similar conditions would result in an image with distorted colours. 

To understand colour constancy it has been assumed that the HVS somehow calculates 
the reflectance, an intrinsic property of a material, at each point in the scene and in this way 
disregards the illuminant. In Buchsbaums spatial processor model [l] a finite dmiensional 
linear model of iHuminance and reflectance is used to estimate first the illuminance at each 
point and from this the reflectance. Gershon and Jepson extended Buchsbaums model such 
that the illuminance and reflectance basis functions are based on statistical measurements of 
naturaUy occurring reflectances and iUuminances [2]. A similar Unear model approach has 
also been proposed by Maloney and Wandell [3]. 

Another approach to this problem is given by Lands Retinex theory [4-6]. Basically in the 
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Retinex theory it is assumed that the HVS performs some random scan of the observed scene 
and in so doing calculates the ratio of radiant energy (reflected light) on both sides of the 
boundary between different objects in the scene. E illuminance is slowly changing then since 
the chosen points are on either side of a boundary they are close together and the inuminance 
at both points will be approximately the same. The ratio of reflected Hght will thus tend to 
the ratio of reflectances. In this way the reflectance at each point can be calculated. This 
theory has been used with some success as part of a neural system for colour constancy [7]. 
It has not gained wide acceptance though both for computational reasons of how to scan a 
scene and how to differentiate between objects and for physiological reasons amcngst them 
being how the%VS could perform this scan, recognition and reflectance calculation in real 



time. 



In this report we do not consider colour constancy. We are rather concerned with how 
detaU and appLent sharpness of a colour image can be improved by manipulating colour. 

A digital colour image is generally represented as three planes such that at each spatial 
coordinate intensities of independent primaries such as red, green and blue are known. Owing 
to this comples three dimensional nature of colour and to the fact that colour perception is 
not well understood, Uttle research has been done on colour image detaU enhancement. In fact 
as Hunt and Kubler note [8], none of the major textbooks on unage processing contain any 
reference to processing of images with a higher degree of spectral complexity than monochrome 
( [9-11]). There seems to be an unspoken agreement that «we can process each of the three 
monochrome images separately and combine the results" [12]. 

Hunt and Kubler themselves approached the problem of image restoration and deblurring 
for multispectral images [8] by developing a technique which is based on optimal (Wiener) 
methods. A further approach to colour image enhancement has been to consider colour spaces 
other than RGB. One of the more natural ways of describing a colour is in terms of what 
colour it is (hue), how deep the colour is (saturation) and how bright the colour is (intensity). 
Thus some effort has been made to enhance in this HSI (hue, intensity, saturation) space. 
Apart from simple global saturation and brightness increases other manipulations have been 
attempted. Lehar and Stevens have presented a fast interactive method for saturation stretch- 
ing [13]. This is done by encoding the image colour data using a peano curve into a much 
reduced set of colours stored in a lookup table. Manipulations are performed on the lookup 
table and so can be interactively viewed. TraditionaUy edge enhancement has been performed 
on the intensity component. Since this component is merely a weighted sum of the R,G and B 
components, the result of linear filtering methods used on it are not much different to applymg 
the same techniques to each of the planes separately. An improvement to this type of approach 
was suggested by Strickland ei al [14]. They noticed that the saturation component contains 
edges wHch are less evident or missing in the intensity image. This change in saturation mdi- 
cates detail which is not evident in the brightness image. To reveal this detail they both edge 
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enhance the intensity component and add to this the edges from the saturation component. 
The resulting image does reveal more detail than when intensity edge enhancement alone is 
used. Another use of HSI space for colour enhancement has been presented by Bockstein [15]. 
Since monochrome histogram modification techniques cannot be appKed to each of the RGB 
planes separately without unacceptably changing the colour balance, he proposed applying 
histogram equalization to the intensity component alone and then equalizing the saturation 
component but as a function of hue. That is, for each band of hues present in the image the 
saturation histogram is calculated and equalized thus maintaining the same hues and only 
varying their^satuxation. Unfortunately results of this method were not presented although 
preliminary icks that we have performed seem to reveal that colour balance is changed and 
that the method is more suited to enhancement for information extraction than for visual 
quaUty improvement. The method is also computationally intensive. 

The shortcomings of aU these techniques are due to a lack of a quantitive definition of 
subjective i^age quaUty. The only way to circumvent this lack is to base the processing 
method on knowledge of the HVS. 

Very simply put, the latest generally accepted way in which the early EVS functions 
is that hght, J(A), enters the eye and is absorbed in the retina by three cone types. The 
responses of these cone types are then transformed to a so caUed opponent colour stage where 
cells are excited by certain colours and inhibited by the opponent colour. The opponent 
processes consist of two chromatic channels, red-green and blue-yeUow, and an achromatic or 
black-white channel. 

Faugeras developed an image processing model based on this visual model [16,1T]. A block 
diagram of his model is shown in figure 1.1. The first stage represented by 5r(A), 5o(A) and 
SaiX) is a linear transformation of the cones and is foUowed by a nonlinear, logarithmic 
operation representing the cones non-linear response. These processed stimulus values R\ 
G* and are then followed by the opponent stage where the chromatic and achromatic 
components are separated. The outputs are the luminosity variable L (black-white) and the 
two chromatic variables Ci (red-green) and C2 (yellow-blue). 

A flaw in this model is that the transform to the opponent space is not known and so 
Faugeras had to somehow estimate it. Using the spectral responses of the cones, Faugeras 
employed the CIE Standard Observer relative luminosity efficiency function V{X) to compute 
a, ^ and 7 under achromatic conditions. Thus the model is only ec^irely correct for achromatic 
images. 

Using this model and Fourier techniques Faugeras showed how dynamic range can be 
improved and how a measure of colour constancy can be obtained. 

In the rest of this report we develop a similar opponent colour model but we use new 
knowledge of the cone response stage and most importantly we base the opponent transform 
on the latest insight into how this is performed in the HVS. The enhancement we propose to 
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Fieure 1 'f. Block diagram of Faugeras image processing model reproduced from [17]. 
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«sc in conjunction with this model is also based on new knowledge of the HVS rarher than on 
Fourier techniques. In particular, new knowledge of the existence of opponent cotour centre- 
suxround retinal receptive fields has led us to develop a colour edge enhancement technique. 

m chapter 2 we give a brief review of biological colour vision which leads on to a discussion 
of the role of the opponent colour transform. In chapter 3 we incorporate the knowledge of 
the HVS discussed in chapter 2 into an opponent colour computational model of the early 
HVS We first describe how the cone responses are obtained from the digital image and th«i 
how the opponent transform is calculated using a Karhunen-Loeve transform. In theory the 
transform is dynamicaUy calculated for each scene but we show that because of the statistical 
properties of certain classes of image, we are able to derive a scene independent transform 
that can be used for any image in that class. We also illustrate some of the properties of 
the produced opponent channels and we relate the responses in these chamiels to the more 
familiar perceptual quantities of hue, saturation and intensity. 

In chapter 4 we develop and describe an enhancement method that may be used on each 
of the opponent channels. This begins with a discussion of the role of lateral iuhibition m 
the tetina and of colour Mach bands. In this chapter we also briefly discuss a measure of 
colour image enhancement that is used to compare results. We then apply the enhancement 
technique to a few images and iUustrate the results. Finally, we discuss some of the effects 
of noise on the model channels and show how chromatic enhancement can be performed even 
in the presence of noise. 
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Chapter 2 

Background to Colour Vision 

colour is a phteomenon in which we visudly perceive some function of the spectral content 
^ hldialt energy emanating from an object. Since radiant energ. ^^^^^ 
Action of wavelength the visual process is a filtering of the input signal ^^ ^^^^^LZ^l 
^sion characterized by a colour name. The details of this process are -t.-^^ und«st-d 
Lthough the workings of the early HVS have been quite thoroughly mvestxgated .18 19]. 
" section w! will discuss some of the findings ^ ^^-\^^'^frS.l:Z 
in particular those that will help understand the compntatW model whxd. we develop 

in the next diapter. 

2.1 Trichromacy 

In about 1800 Thomas Young suggested that physiological colour processing in --*J>« 
^strilted to the involvement of three independent variables. In 1866 Heh^olt. pubUshed 
trh^othtis that there exist three different channels with three different fi^t^s and even 
^op'?erthe spectral response curves of these filters. Later Maxwell provided quantxtave 
tion ofthe so called Young-Helmholt. trichromatic ^^^^^^ J^^^^lr^ZZ 

cordirming the trichromatic theory includes that of Bnchsbaum and ^^^f^^J^^^!^ 
that practical cows are either suitably frequency limited to be represented by three samples 

or have suitably frequency limited xnetamers [20]. 

pro«L of ft. t^.. ^ cc«. Th.s. m turn ar. du. to ft. ftr« ph=- 

C^L! bund in »ch ot ft. con. .rp.s. By ».««ing ft. .p.c..1 "t 

3r o>n.d l^d, G,«. ..d ««. b.. fti. is n.i...adins .in., .to. n«ft^ 
r. cr b. ft. colo» ».od.»d wift ft. s».i.ivi.y c". not of ft. ph0.0p.5m.nt. In .ta 
prrf» « n,, .h. n»n,s lon^ (1), m.di™ (m) .»d (.) w.v.l..»ft r,spo„s.. 
Zt.»pU of ft. con. r^pon^ cnr,» „. fto» in ii^ 2.1. In .pp»d.x A w. b„.fly 
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Figure 2.1: Vos-Walraven cone spectral sensitivity functions Kn(A), and ^^(A) [18] 



discuss how the trichromatic theory is used for colour naining and in digital images. 

2.2 Colour Opponency 

There are many colour phenomena which the the simple trichromatic the«y cannot exp W 
For example ahove 600 nm the short wavelength cones are ineffective: We therefore would 
expect that stimulation above this wavelength- would appear greenish red when m fact it 
appears yellowisli red or orange. 

some of these problems were recognized in 1878 by Ewald Henng who P"P-<i 
temate theory for the perception of colour called the opponent-colour theory l^H- Henng 
proposed three mechanisms that mediate in colour vision: one that accounts for the p^- 
LpL of red-green, the second accounts for hlue-yellow and the third for black and white 
diLctioiis. This theory was not widely acceptedparticularly with the physiolopcaldxs^^^ 

of the three cone types. .1,1. v 

Relatively recently at least four varieties of so-called spectraUy opponent cells have been 
found in the LGN (lateral geniculate cortex) [16.19]. These cells have shown exatatxon to some 
wavelengths and inhibition to others when the eye was stimulated by flashes of monochro- 
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matic lights. Moreover this tj-pe of response was maintained over wide intensity ranges thus 
indicating that the cells responded to colour rather than luminance. The cells whose re- 
sponses showed maximum excitation around 500 nm and maximum inhibition around 630 nm 
were called green excitatory, red inhibitory cells (+G-R). Other ceUs showing rough miixor 
image responses were called red excitatory, green inhibitory (+II-G). The cells showing max- 
imum excitation (inhibition) at 600 nm and maximum inhibition (excitation) at 440 nm were 
named yellow excitatory, blue inhibitory ceUs (-hY-B) and blue excitatory, yellow inhibitory 

cells (-1-B-Y) respectively. 

In addition to these four varieties of spectrally opponent cells , two other classes of cells 
were found thlt did not give spectrally opponent responses but rather responded in the same 
direction (either excitation or inhibition) to lights of all wavelengths. These cells thus appear 
to be involve^with achromatic vision and are thus named white excitatory, black inhibitory 

(4-W-B) cells, and vice versa. 

More recently sHghtly more complex opponent cells have been found, both m the final 
retinal processing layer, the retinal gangHon layer as well as in an earlier retinal stage con- 
taining the hfirizontal cells [18,22]. In addition to the weU known intensity (black-white) 
opponent centre-surxound cells four types of colour opponent centre-surround ceDs have been 
documented. These are red centre, green surround and its mirror image as well as a blue 
centre yellow surround and its mirror image. 

Thus it seems that the Young-Helmholtz trichromatic and Hering opponent theory are 
complementary; the first dealing with early retinal processing and the second modebng phe- 
nomena at later stages of the retina and at the LGN level. 

2.3 Opponent Colour Transform 

Although we know that radiant energy is mapped into three independent variables by the cone 
responses and later into three opponent channels, the transform from the cone trichromatic 
responses to the opponent colour responses of the retinal ganglion ceUs is still not clear. 

Many psychophysically based models of this opponent process have been proposed though 
Uttle physiological backing has been found [16-18,23]. 

Recently though, by considering the HVS as an efficient information processing system, 
some surprising results have emerged. What we mean by "efficient" is that redundant informa- 
tion is reduced and energy compressed such that the chamiel capacity required to transmit the 
information at a given level of reUability is minimised. Reasons for considering this approach 
in the HVS are that natural visual scenes exhibit both spatial and chromatic regularities, 
making the retinal image contain much redundancy. The ratio of photoreceptors to optic 
nerve fibres is at best 10 : 1 if we only consider cones and about 150 : 1 if we take rods mto 
account as weU [22]. Thus if the initial photoreceptor responses were left unaltered, the hm- 
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ited channel capacity of the nerve fibres leading to higher stages of the HVS would be miable 
to cope. This reasoning has led to the hypothesis that the purpose of image transformation 
in the early visual system is to reduce correlation and compress information prior to neural 
transmission. 

Based on this hypothesis, Srinivasan ei al [24] accounted for the spatial profile of center- 
surround retinal ganglion ceH receptive fields using a linear predictive filter. They showed 
that the spatially antagonistic centre-surround structure of these receptive fields decorrelate 
and suppress an image signal. As such they can be viewed as method for removing redundant 
information arising from the high spatial correlation of natural scenes. 

Using sim^ reasoning Tsukamoto et al [25] demonstrated that a gaussian spatial profile 
of centre-surround receptive fields is optimal in terms of spatial correlation reduction. The 
profile of such retinal receptive fields are indeed a close approximation to being gaussiai. 

Apart from the chromatic regularities of natural scenes, additional chromatic corrdation 
is due to the highly overlapping nature of the cone chromatic response functions as can be 
seen in figure '2.1. In- a study of this, Buchsbaum and Gottschalk [26] used an eigenvector 
analysis to in'Testigate the role of opponent type processing in colour vision as well as the 
relation between opponent colour transforms and the initial cone responses. They showed 
that redundancy reduction and information compression is achieved by a transformation to 
an achromatic and two opponent chromatic channels. Thus the opponent transform can 
be viewed as method for removing redundant information arising from the high chromatic 

correlation of natural scenes. 

Denico and Buchsbaum [27] have combined these two approaches of spatial and chromatic 
correlation reduction into a computational model of image coding in early vision. The first 
stage of their model consists of performing an eigenvector transformation on the original 
red and green planes to reduce chromatic correlation. They then use linear predictive coding 
(LPC) on the achromatic channel alone to reduce spatial correlation. By applying this coding 
technique to only the red and green planes of a small group of images they transform an image 
into two channels whidi are Icnown to have correlates in visual system psychophysics. 

What we can conclude is that treating the HVS as an information processing system is 
a good approach. Also as regards the opponent transform, it seems that by transforming 
the cone responses in such a way that redundancy is reduced and energy is compressed, we 
should be approximating very well the operation that the HVS performs when it transfornxs 
from photoreceptor to opponent responses. 

2.4 Contrast Sensitivity 

Although the transformation to the opponent channels has been quite unclear to this date, 
some of the properties of these channels have been psychophysically investigated. In partic- 
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ular v.e are interested in the difFerer^t in which lununance and chrominance frequency 
i^orxnation is perceived. Various models for luminance and chrominance -trast s^.t.v,^ 
^.ctions fCSF) have been proposed, for example [28]. AJl studies agree on one pomt. That 

Ilatic contrast. In fact above a spatial fre^ency of about 3 cycles/degree ^e HVS^ 
i^ensitive to chromatic contrast while it is stiH quite sensitive to Immnance contrast. Tlus 
Zi^c. in sensitivity is particularly relevant when high frequency ^"^^^^ ^ 
ered It should be noted that the blue-yellow chromatic channel « even less sensitive than the 
^d-green channel to chromatic contrast. An example of typical luminance and chrommance 
contrast sensitivity functions are shown in figure 2.2. 




-t.i 
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channel is less than the red-green channel. 
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Chapter 3 

Development of Colour Image 
Enhancement Computational 
Model 

3.1 lYansforxnatiozi of RGB to Cone Primaries 

As described ia section 2.1 tberc are three cone types inrolved m htmaa colour Tision tadi 
of which co&taizis a pigment of different spectral absorption characteristics. Based on the 
trichromatic theory, colours maybe thonght as being represented at the first stage of the retina 
by the tristimnlus valnes derired from these absoxptioQ enrres. These enrrts are tened 1S(A), 
Kn(A), and T^(A), where f,m,s stand for the long» medium and short wavelength responses 
as preriottsly discussed (section 2.1). Colonrs in a digital image, on the ether hand, are 
comprised of the tristimulus values derived from the spectral functions of the camera sensors. 
These are commonly called Red, Green and Blue owing to the location along the wavelength 
axis of the respective peaks of these spectral functions. The location of these pesls imply for 
example that an image with KGBs(0,0,l) will look entirely blue. 

ThcO'e are two reasons why we require that digital colour images be represented by the 
retinal tristimulus values: 

• We wish to use the model to enhance colour images for the benefit of a human viewer. 
Therefore we would like to exploit characteristics of the EV5. For example in adiromatie 
images we increase luminance detail by enhancing luminance edges. This is done on the 
achromatic image itself since it represents luminance. In the same way we would like to 
perform enhancements on colour images in the EYS colour space. 

• If we assume that the HVS transforms to the opponent*colour space using some type 
of information transmission analysis on the cone responses, then performing such an 
analysis on other tristimuli will not guarantee that the resultant space is similar to the 

37 



SUBSTITUTE SHEET (RULE 2^ 



wo 94/11987 



PCr/US93/11146 



HV5 epponezit space. 

Henet the fint stage of the model mrui be to trsjuform the image represented bj the EGB 
set of primaries to the EV5 set of primaries. 

It is known from Grassmans' laws that the EV5 spectral absorption azrres mujt be a linear 
combination of the S(A), JI{X) and 2(A) esms, deriTed bj the CZE from colour matching 
experiments [19]. However the particular linear combination is not known. Ihns in the 
literature th« exist a anaaber of "fundamcntaT qrstms [18). The two most widely aeecpted 
sets of fundamentals are those of Smith and Pobornj and of Vos and Walraren [18]. These 
two sets of spectral absorption eurrcs art vey similar. The main differnee Is the rtUtiTt 
response of the short wavelength response function T^(A). Bn^baum has shown that if we 
fogjQ^ opponent diaancls following information transmission efficiency considerations then lisr 
a large raag^ of relative responses of T^(A)t the resulting channels are independent of the 
a rt^^i rtsponie [26]. Since this is the basts of otff aodelt the choiee between these two sets ef 
eurvu is not:eritiei] and we have arbitrarily chosn to nse the Vos«WaIravca set. These are 
illustrated in^figure ill. 

By the tri^omatie theoryt if the fundameotals in a set of primary colours are linearly 
independent, then it is possible to derive a linear transformation from a colour represented 
by one set of primaries to a rcpruentation by another set [18,29]. We wish to £nd the S x 3 
transformation matrix, Tjtv. that transforms RGB tristimuli to ViV^V,. That is, 







'it' 






G 






B 



The transformations from RGB to XY2, Tax [30], as wcH as from VJV^V; to Z?2, Tyx ll«» 
30] are linear. The transfonnation from to XYZ which may be implemented using the 
Vos equations [31], is a non*linear function of the luzoinanee and is constant oaly if y is 
unchanging. Rather than derive the transformation for each pixel in th« colour image m 
have assumed that for image processing purposes, the XT' 2 and XYZ values are sufficiently 
dose that this traniformation is not necessary. This approach is verified by Guth who has 
used the same approximation in his latest model of colour vision [23]. We have adopted this 
simpliiication since it accelerates the transforxoation. This is important if the model is to be 
used for resl»time applications. 

With this simplification in mind, using matrix algebra we get, 

TRv^^Tyxr^^Tnx (3.2) 
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which results in, 



Txv 



2J6S3e-Dl 3.8702e-01 7.9«7e-0l 
4J417e-01 2^035e-01 1.7708* -01 
3.6682e-08 2.4582e-04 4.U04e-08 



(3.3) 



Thus at each .patiel loc»tic« in the image we t»«sfbnn the MS tristiadi to W^V, uing 
the colour tn&sfotm. Tjtv. 

S.2 Sep^ation of Chromatic and Adxroxaatic Information 

The tocwBoa of .ectioa iJi h« led « to Wiew that ih» Mppra^ cS ensideriiig the 
HVS « « efcJent ii^omatioB proeeoing .y«em U . food on. ttd that the aort «t«l 
tr-ufonoatS eoloux i^uge. i. «. that will mdt in the hut 

energy compression. The c^tia« tr«»«for»ation in term of thue re,«r«^ to^ 
Kaihanen-Loeve traisfoia (HT). Tie QT i. pptiad linee it «a be Aow» th^ tit ««r 
between the 'original i»«8« « «co«tn,eted from a ,rop«ly '^^^^V^ 
KIT component, ii the nnalleit .mongn .H Ibear image tr««f«m. a^*" 
the next step afte transfonning from iiffS to V,r«V.. i. to apply the laT to 
V„ V„ and", cone responsu at each .patial coordinate, the two ..cUo« w. «^ 
how the Kaihonen-Loive tranrfom is ealeulated. 

5/2.1 Discrete Karhunen-Loive li-tflsform 

This transfonn .Iso known a. the principal component, ^genrector er EoteHing tr«ufonn 

can be formulated a* follows: j « s. w 

If {.^(n). l<n<Jt}U* random aevene. with autocorrelation matra £ and « » an JtT 
by unitary' matrix that reduces R to dinonal form then the transformed rector 

2^§X M 

is called the discrete K.xhun.n-Lo*ve transfom of Jf. It can be .hown that the tow, of 
* are the nonnalired elgm^ectcr. of Jt 19.10.32-361 «d that these «g«»vector. «. .^o 
orthonormal 137]. Therefore the dements of the ..^enc. 7(« «. "^^r^-^J* /^l^^^ 
.ho*-n that if « is made up of the normdi.ed dgenvector. of the cov.r..nee aatra C rather 
hroft: Icorrelat J matrix K. then the sequence ^(n, wIU also he -i"-^^-^^; 
cev-arianc. matrix is simply the corrdation matrix of a aero mean process or dt«at,vdy. 
the correlation matrix of a non-zero mean process that has been centrd^red. In oth« word^. 
R(„.r„) = and C(n.m) = - - m.] where m. » the mean vdue of 

'A »r.itMy ««:jx « o«. .«b ih.t iu tq«Ji lu ee«}«|»i. U«.p.«d. U. X"' - [A'f- 
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3.2.2 Multispectral Discrete Karhunen-Loeve Transform 

We now adapt the discrete Karh^en-Loeve transform to the multispectral case In a multi- 

spectral image, a scene consists of M spectral planes. In the case of KGB,M = Z^ 

lit X(i /) represent the intensity vector whose Jlf elements are the values of the p:xels 
at location (Jb,/) in the set of spectral planes. That is, 



X(i:,/) = 



X2 



where 



Jb = 1,2,. 
/ = 1,2,. ..,W 
Xi = intensity at (U) for plane i, and 

X and W are the number of pixels per row and column respectively, of each spectral plane. 

XfJfe I) is the random vector whose components we wish to decorrelate. Since there axe 
X . such vectors, under the assumption of stationarity and ergodicity we may estimate the 
covariance matrix as shown below. Note that the assumption of stationarity is not m general 
entirely true because of the changing spatial characteristics of natural images. 

The covariance matrix of dimension Af x Af , is defined as, 

Cx = i?l(X-mx)(X-mx)'] 

where 

mx = E[x] 

It can be approximated under the above assumptions, from the L-W samples [9,38], by, 

Cx « 7^ EETOA./)- ni.KX(i.O-xn.n (3-5) 

Using the image data in expression 3.5 then results m the M X Jlf covariance matrix 
C Since M is usually small, it is a relatively simple matter to find the eigenvectors and 
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eigenv^ues of C If we denote the normaH.ed eigenvectors as ^ and the corresponding 
eigenvalues by A„ then the transform matrix f is: 



where is the j'th element of the i'th normalized eigenvector. 
Eigenvectors are normalized in the usual way, 



That is, the rows of * are the normalized eigenvectors of C By ordering the rows of 
* such that.^e corresponding A„ are in descending order, the energies of the resultmg 
transformed unages will also be in descending order. , 

It can easily be shown that the covariance C., of the transformation Z(A,/) = *X(A,i) 
is equal to *£!x*"^. ^ can also be shown that: 



Ai 0 

0 



0 
0 



0 0 0 Xm . 



(3.7) 



wwe A > A, > > Aj^ are the eigenvalues of C,, Thus, the transformation produces 
^aUstitlly decorr;iated images, at least in the sense of having a diagonal covariance matrix 
To sunLarize, we transform to the opponent space by using expression 3.5 to estimate 
the covariance matrix of the cone response data. We then calculate the eigenvectors of this 
Itr^, and arrange the normalized eigenvectors such that the respective eigenvalues are m 
decreasing order. This then gives us the transformation,*, to the opponent space 

We then use $ to transform each colour vector at every spatial location m the image. 
This results in a rotation of the Vu and V. axes, forxning a new set of orthogonal and 
Icorrdated axes wMch we label iT, , iT, and ^3 that span the colour space. Unhie a prev^^^^ 

study [26], the resulting channels have a spatial structure whose nature we exploit not for 
coding purposes but for enhancement. 
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3.3 The Enhancement Model and Its Relation to Other Mod- 
els 

Now tiat we have reprweat.d the digitil eolooi image in a «paee eloiely allied to that the HVS 
„es, we perfoim eJuaetmeat oa taeh of the Kt ehumdi. Th« letttil typt of .nhaBeeBunt 
we KJMider is ea^laiaed tljewhere (lection 4.2). To Tiew the ntaiXt we hare to iaTene 
traojfom bade to RGB. It i« dear that {Tkv)'^ oxUti but we alto hare to be me that 
f -1 existi. Fortimately, be»uc C, ii real and iTnaaetric, asd therefore it» Bgeavecton are 
orthononnal (9], we h»Te that: 

The iavene thu» always «Bttt a&d iawse traasfoimatioo is alwayi possible. 

We are aow i:^ position to draw a Ueeh diagram of the BodeL This is shown in figare Si. 

JflX 




Figure 3.1: Block diagram of the generalized computational model 

Tfon awrnn psnn maw rararr :3.1 



Since C. is real and positiye it will always hart crdy one eigenvector whose entriu are 
of like sign [26]. This means that only one of the transformed channel* generated fcy ♦ e«i 
be an positive combination of V,. and V,. The other two channel, must therefore be 
antagonistic. The all positive channel dearly contains the greatest proportion of the Image 
energy and is assodated with the largest dgenvalue. ThU diannd is Ubdled Kx. The two 
other aatagoniJtie channels are labelled Kt and Kt- 

We show in section 3.4 that the wavdength response function of the K^ d.annd approx- 
imates wdl the HVS luminous effideacy function. We therdore call this the adaomatic 
dianad We also show that the wavdeagth respoase fimctioas of the JT, aad ft diaaadi 
are simUar to the response ftnetioa. of the opponent red-greea and blue-ydlew diaands 
diophysicdly found in the SVS [IB.IS). We therdbre dassify oar modd as an opponent colour 



42 



SUBSHTUTE SHEET (RUII 26) 



wo 94/11987 



PCT/US93/11146 



model along the lines of the theory fint proposed by Bering [21]. Many attempts have been 
made to quantify his theory, the most recent of which being by Cuth [23]. Faogeras [I6|17jhas 
been the only one however to attempt to use an opponent colour model for image enhancement 
purposes. 

The main diiFerenee between this model and Faugeras* is that we dynamically transform 
to the opponent channels for each scene based on an emulation of the behavior of the HVS. 
Faugeras transforms nsing a constant transforniation that he derived from psy^pphysical 
experiments. Jn section 3.4 we describe an experiment that shows that for scenes belonging 
to a similar class onr model may be reduced to a constant transform. We may thas consider 
it to be a m^e generalised image processing opponent colour model. 

Another diiference between oar model and Fangeras' is that he transforms the logarithm 
of the cone responses. Thus for blade and white images, his model rednecs to*Stoeieham*s 
homomorph^ model for adkromatic visioxu Stodcham proposed a multipHeativt homomosphic 
model for processing achroznatic images that matched both the structur e of images and the 
EVS [39]. The multiplicative process of image formation where an achromatic ixnage i((s,|r), 
is the product of an Uluzninance function {(s.y. A) and a reflectance function r(s,|f,A) can 
be undone By a logarithmic operation. This is because the logarithm is a homemesphic 
function from multiplication to addition. Eence it maps the image product into the sun of 
two components which can then be separated by iinear filtering since in general the frequency 
content of s*(«,|f. A) and A) is distinct. StocLham*s model also matches the EVS sinee 
the cone responses are nen linear for very high and very low levels of illumination. His modd 
is espedally useful for improving dynamic range. 

Faugeras argued that as the frequency content of the luminance component that is ass> 
dated with the illuminant is different to the reflectancCt then so should the frequency content 
of the colour components, which are associated with the colour of the Uluminant, be different. 
Faugeras thus also used a logarithmic non*linearity to achieve dynamic range enhancement in 
his luminance channel and a measure of colour constancy in the chromatic channels. Because 
of the three dimensional nature of colour vision however, a colour image cannot be strictly 
considered as simply a product of illximinanee and reflectance fimctions. Colour image forma- 
tion is a multiplicative process that yidds the product of i(«,|rt A) and ^(e,sr,A] but before 
the visual system (EVS or camera) processes this product, the light is linearly absorbed by 
three sensors or cone types as expressed by equations A.S. We no longer have a loga- 
rithm of a product but the logarithm of a sum of products. Although it has been shown that 
under certain circumstances it may be possible to approximate the image tristimulus values 
as products of illuminance and reflectance tristimuli [40] this is in general not true. Also, 
while it is true that the cone response is approximately logarithmic, we are not sure what- 
other non-linearities there are prior to transformation. For these reasons, we have dected not 
to use a hcniomorphic model and not to indude the logarithmic non*Iinearity at this stage. 
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Ai t ftsal eemme&t, wc Imow iknt the problem of colour eoastkncy is f«r more complex 
than FMgeru' model tuggMts. Wt also know that if desired, dynamic range eas be improred 
by poit-processiag en R', G' and B' usiag a logaritimie sealing meAod as raggestad by 
Pratt [34]. Thus, in terns of image proeesiing, taking the logarithm of the cone responses is 
not of mttdi benefit. 

3.4 Scene Independent Transform 

Up till now, aS opponent eoloar models hare been based on a constant traufocm from cone 
responses to iromatie channels Il»-18,23]. This transformation has been derired by fitting 
model parameters to Tarions seu of psyehophyrical data. This Is an aeetptable appteaei if the 
models so derrved are to be wed only for predicting psychophysical responses. When the vissal 
stimulus is ajore complex than sinusoidal gratings or edonred patches, we cannot be certain 
that theie models aeeurately reflect the beharior of the HVS. HoweTer, it is reasonable for a 
number of reasons to assume that the opponent tranafoxmis constant. Firstly, the transfbsmis 
from photoreceptors to ganglion call rtspenses and is thus pefoimed entirely in the retina and 
secondly, the transform is performed in real time. This b net to say that the colour channels 
of tbe HVS are non-adaptire but that adaption to a partienlar scene and ffluminantis gradnaL 
Adaption aids in our ability to recognise colours under many different Uluminatien eonditioM 
(colour constancy) but we assume that under any one partienlar eenditien of illaaiBatioathe 
colour transform is constant. Adaption is also assumed to be performed later en in the HYS 
and not in the retina. 

In what follows we show how our model, which is able to take into account the complex 
nature of scenes, because of the use of the Xarhunen-to<Te transform, can also be need to 
derive a constant transform for complex scanu of n similar qrpe. 

Sued en the assumption that man hu adapted to his environment, it is reasonable 
to assume that the HVS hM optimally adapted to scenes which it most often encounters. 
Such colour scenes are natural daylight type images. If this is the ease then the opponent 
colour transform should maximally deeorrelate and compress such images. If we consider Oe 
converse of this, then maximal compression and decorrdation of natural imagu should yield 
the opponent transform present in the HVS. 

b this section we have esperimentally invutigated this idea using a real set of natural 
images. Seven colour images of various natural outdoor scenes, similar to those showa in 
section 4.4. were used to derive the constant transform. They were taken with difTerint 
cameras and under different dayBght muminaUon conditions. la this way we hoped to obtain 
as good a representative ensemble of images of this type as possible. It must be emphasised 
though that although the set of images consists of a large number of triplets of red, green and 
blue pixels, this in itself is no proof that the images we chose are representative of the entire 
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eaiemble of a&tural imagej. Oar nsaltt do tteni to suggest howcrer that these images are a 
reuon&bly good representation. 

To derire the eeastaat traasfona, we first hare to estimate the efaromatie eoTiriaaee matrix 
of this set of images. Since spatial eerrelatiees tr* of so eenseqaeaee for this, one method is 
to simpl7 form a single n^efset consisting af all the snilabU eoibnr txxplcts. This super-set 
eonld then be used to estiaaate the colour eovariance. The problem with this method if that 
owing to the immense amount of data (7 x 512 x 512 x 3 a S.25 m for 7 images of resolution 
812 X 512), it if set feasible to use directlj txprwioas 3.5 and 3.6. Ibr this reason, we* derived 
- an estimate of the cervianec of a set of data based seldr en the means and cemiances of 
each image £ the set. Details of this derivation may be finmd in Appendix B. The dcriv»tiaa 
resulted ia, 



7 E -ffff-- 



where 



■)(!-)] 



(*.»} 



n m 1,2,. 

N m Bimiber of images m ih^ set. 
Cn « eomiuce matrix of the A'tHimase. 
BBn « mean Teetoroftheix'tli image. 

C s covarianeematrizof the set of images. 

Applying equation 3.9 to the set of images and ealoilating the noxmaliied eigenvectors 
resulted in, * 

0.8517 0.3070 0.0032] 
<< « 0.3068 -0.9516 0.0193 I (3.10) 
-0.0089 0.0174 0.9998 J 

Buchsbaum theortticdly Tcrified that maximal decoirelation and compression predict 
an opponent colour transform similar to that psychophysicallj found in the HVS [26]. fit 
did this by assuming that the image, as a function of wavelength, that reaches our eye iae- 
longs to an ensemble of images which have a broadband Fourier frequency power spectrum.* 
This is true only for images made up of monochromatic signals such as those used in visual 
psychophysics. The constant transform which we have found using natural images, not 
necessarily made up of monochromatic colours, is similar to the transformation matrix that 
Buchsbaum theoretically calculated. This result reinforces Buchsbaums conclusion that the 
goal of the opponent transform in the retina is to achieve redundancy reduction and infor- 
mation compression. It also shows that the HVS seems to be optimally adapted to natural 



scenes. 
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To iUustme that our aodil u indeed u opponent colour model ud tli»t the colour 
space we traaifona to ij limilar to the EVS colour space, we use i, to tranaform the Voi- 
Walravea curves shows m fifure 2.1. This xtsults ia the efareaatic response functions shown 
ia figure 3.2. Ja figure 3 J (a) «c cu see the dose agrtimcat between tht model predicted JT, 
channel and the relative lumiaons effidencj curves derived by Jadd (ISJ. Also, the K» aad JTa 
chromatic response functions shown in figure 3.2 (b) aad figure 3.2 (e) respectively, are simaar 
to the red-green and blue-yellow opponent colour ehaaaels derived by various ruearehers [18]. 

The impUeatioa of being able to derive a constant tnasfbzm of* set of images ia tctms of 
image processing is that by using our model to derive eonstaat traasforms for differeat'dasses 
of images we eaa emulate a visual system that optimally adapts to the set of kuages with 
whidi it de^. Wlat this meaas is that the izpeasivc and time eoasumiag pre-processing 
stage of deriving « caa be performed efBiae. The traasfennatioa to the eppoacat colour space 
eaa then be performed ia real time by two matrix traasfoxaaatioai ij simply ideati^iag the 
scene type and using the relevant pre-detei^cd f . The geaeral swdel of figure 3.1 can thea 
be reduced Co the simplified model of figure 3.3. 

Ia sectioa 4.8 we coaqiare tht results of pcrfbzmiag the same eahaaceacat ea «a iauge 
transformed first aceordiag to the gcaeralised mode! of figurt 3.1 aad thea by the simpliJied 
model of figure 3.3 which uses the eeastaat traasfeim. Visually the difTcrtace bttwaea 
the two ruults is imperceptible. 

3.5 Frequency Properties of Karhunen-Lo^ve Channels 

Since we have used the Kariuaen-Loive transform to obtain th% opponent ehaaaels aad 
because JT, is associated with the largest eigeavalue, Ki wiH contain a significant portion of 
the total image energy. Kt is associated with the next largest eigeavalue aad JTj with the 
aezt, so they wiU respectively eoataia less caeigy. 

To illustrate this we have transformed two typical RGB colour images to the JT span. 
These images are shown in figure 4.4 and figure 4.13. We then estimated the percentage 
energy ia each channel by talcing ratios of the eigenvalues estimated by the Karhuaen-lolve 
transform (i>- = Ai/(Ai + A, + As)). This method caa be understood riace cadi eigeavalue 
A.- is e<iual to the variance of Ki aloag eigenvector it [9], as expressed by cquatiea 3.7. The 
results of these ceJcuIations are shown ia tabic 3.1. For eomparisoa we have also shews 
the wiounts of energy that are present in the original red. greea and blue diaanels. The 
distribution of energj- in the transformed images (table 3.1). shows tliat practically all the 
energy is contained in the adirematie image {Kt) while almost ae energy remains in the 
red-green (Ki) or blue-yeDow (JiT,) diromatic images. This is a result typical of the use of 
the Karhunen-Loeve transform on chromatic images [8,34,38.41j. 
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Figure 3.2: Spectral sen^itiTitj of tbe adirom&tie (JTx) ebazmel compared to Jtiddi rdatire 
Itazninotti eSdeney curre is shown in (a). Shown in (b) and (e) are the spectral sensitivity 
curves of the red-green (Kt) and blueojdlow (JTa) channels. The Ki curve has been shifted 
upwards for comparison and the Kt and K9 curves arc normalised. 

..(Jndd)wj ^ mjrru mra nmiar? rmsm (JTi) 'ODrnm yrm * to jrw^ smm m :3.2 Trie 
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Figurt 3.3: Ble^ diagram of the sispliiicd eemputatiesal model 
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la. previous studies 18,34,88,41], where the Karhwun-Ioire traBifeim wm used for eediag 
purposes, the origincl RGB Talues were Karhuaea.to*Te trMufenaed. The taergy disttibu. 
tion in the traMfoimed pliaee in these easu were of the order of Ki : JTi : Jfj « 86 1 IS : 1. 
The reason that we hare aehiered ruh a large eoaeaatration of anergy la the Ki ehaaad rda- 
tive to the other two channels is baeaase we hav« transformed the cone tristimuli, Vi, V„ and 
r, and not the origiasl RGB wlues. looKag at ilgura 2.1 we set that the amplitude of V;(A) 
relative to V5(A) and V^{X) i% much smafler. Hence there is already a measure of information 
compression prior to tr«isforming. ff we consider the viewpoint that the Karhanan-lo^vt 
traasfonais merely a repositioning of axes so as to achieve minimum variance along each axis, 
then it U dear that because along one axis there U already small relative viriaaee then one of 
the transformed axes should be parallel or nearly parallel to the uatransformed V, axis. This 
is indeed the case aa can be seen from the expression for «, (expression 3.10). Therefore it 
seems that the cone response feaetions have also adapted in the HVS inch that when fdlowtd 
by the opponent stage, energy eompressioB wifl bt maximiaed. This result Is in keeping with 
the role of the HVS as an efficient information transmission systcaL 

To understand the difference between performing the KIT in the RGB sabspaee or ia the 
cone response space ViV„V, it must be noted that the RGB colour space of digital eolbur 
images is not an orthogonal Cartesian snbspaea of Ji» in the mathematicel sense. That 
is, the RGB subspaee is not spanned by mutually orthogonal vectors or in other words, 
the spanning vectors of the RGB subspaee are inclined at some indeterminate angle to one 
another. Perceptually however Red, Green and Blue tend to be considered as independent 
quaatitiei which is why in colour science they axe traditionally represented on orthogonal axes 
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U&ding to tooU rich u the colour cute. ^ v w 

Since RGB it not u orthogonal fabsp»ee the tr«ufonn»tion fcom RGB to the V{V^V, 
„b,p»e. c«mot bt Tiewd u a daple tr».Utic«i, toUtien .cling of «e.. Cowidenng 
r«v m. » linetf tr«ii&r« of A» that Bupt » «b.«t to another nbset ViV^V,, then 

ft would be a staple matter to calcolat. the .p«ming rector, of ViV^V. bom the panning 
rector, of RGB [42]. Since we do not know which rector, .pan RGB m, 
Jd that w« can ctimate at thi. .tagt i. that «d« the .taadard B««Ld.a« metne of ^ 
deaeU. of «:b.p«e V,V^V. wifl b. dom tog.th« along the V. «U than they were dong«7 
diaennon of the RGB wbipace. Thi. can be mn from the transformation Tnv (3.3). Tha 
•ffectirdy wLp. the pixd doud no that r«ianc. i. ttduced along one dimension prodnang . 
more elliptical cloud. The proce.. of fitting orthogonal axes to this data nch thai eotrdatwa 
i, aiinimised along each axis (IttT) i. then Amplified. 

B.ca»a^jri is caleaUted a. a wdghted «« of th. poiitire V,. V« and V, ttiitia^ 
edge. «id Ji otl« ,patialiBfonn.tioai. pteserred in thi. AaaneL We can tha. think of 
as a spatialbandpai. eh«md ai it should contain q»atid frequeade. from BC up to .ome 
high frequency. Xt «nd Kt are computed a. wdghted differenctt lad htnct Ugh frequtncy 
Joraatioa Aould be reduced. Thi. I. b.c««« tdg.. la th« «ff S « W^V. 
«udly at the .ame q>atid location. Subtraetioft of one plane from another thus reduce, 
the amplitude of the edge, and hence the amount of high frequency information. Spatially 
therefore we can coadder them to be lowpas. diannd.. To mu.trat. thi. we calculated the 
average power .pectrd dendty (PSD) £br the -rieW image of 4.4. The PSD w«i 
calculated using the dassieal spectral estimation method of periodegram areragmg 112,43]. 
The two dimensional estimate was then areraged over eadi radial frequency to obtain a one 
dimensional represeaution- . . ^ , 

Shown in figure 3.4 U the PSD of the Kx and Kt chaaads. The PSD of the K» diaaad 
i, not shown as it i. dmost .ero for all frequende. except for thoee do.e to DC. The high 
frequency nature of the Ki plane and the low frequency nature of the Kt (ind K») plant » 
evident. The low pas. nature of the dtfomatie dunnd. become, dgnificant when we cenndar 
enhancing noisy image, (.ectiea 4.6). 

In figure 3.5 the amplitude .pectrum, whidi is the square root of the PSD. U .hown on a 
logarithmic .cale for all three diannds. It is interesting to note that the bdianour of th* Ki 
Aaaad i. approximatdy of the form of 1// on a logarithmic .cale. Thi. i. in accordance wjth 

what Field ha. found for naturd adiromatic taage. [44]. The bdiariour of the diromatic 
ehaimeli ea the other hand i. diflTexeat. If we ignore the dc region of Kt then both K, and JT, 
faU off at a rate of approximatdy 1/2/. Thu. it .eem. that naturd image, not only exhibit 
chaiacteristic behaviour in term, of thdr luminknce component but d<o in term, of their 
chroniiaance components* 

It is d$o interesting to compare the spatial diaracteristics expertaentally found with the 
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Figve 3.4: Arerage logtritlmie power fptetnm of tht JTi aad JTt plants. Ki rttuns a 
large proportion of the total image energy at high frequencies wlule JC2 (and K^) contains 
significant energy only for low frequencies. ILe frequent axis raits are in ^cles/pietnre 
where each picture contains 356 x 3S6 pixels. 

jrruH 7SD rm jtott jtto rto JSTj trnuj tmru JTj-i Ki ^mc Tunuft peon cnoisso is A ttm 

S^TOirU 256 X 256 nV'JQ TWOTi 

psyehophysicallj predicted aodulation transfer function shown in figure 2.2. la the EVS, the 
lumiunee channel is indeed bandpass while the two diromi^tle channels are lowpass. Miis 
lends weight to the observation that the way we calculate the opponent transfonn results in 
a colour space siznilar to that of the EVS. 

Before we leave this section it is important to note that although relative to the achromatic 
JCi channel, the chromatic channels are low pass, as channels within themselves, there is still 
a significant amount of energy present at high frequencies. 

3.6 Perceptual Interpretation of K Space 

One of the more natural and familiar ways of describing a colour image is in terms of hue, 
saturation and intensity (SSI). We would thus like, at least roughly, to relate the responsu 
in the K channels to the perceptual quantities of SSL To do this, we use the constant 
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It^'X^''^'"'*^' *mplit»de .p.etnmi of the iTi, JT, and JT, pl«nei rq>ru«Bt«d m leg-lof 
«6 ; »6 p^S"'' ^ Mnwliied cydtt/pietar. whcr. «ch piet«. eonfidJ 

wm-a jsw TT/n t» jiTunms to » an Jir,-i JT, jSTj mte^ msun :3.B iru 

ai'wKn 2$6 X 256 rSOT nnan la jumra mm im nmrao 



tr«i«fena », tii»t was derived in seetioa 3.4. Ihi» i* jtu tifiable mee is witable for • wide 
cl«, of images titat we ecmsider tlie HVS Las optimally adapted to aad because it is the tn>e 
of image that we maialy consider in this stuij. 

Rom figure 5.2(a) we caa see the close agreement between the model predicted JT, ehaaael 
^d the relative luminous efficiency curves derived by J«dd (ISJ. It is dear from this that 
JCi contains the lumin«ieeinformatie»whidiiswlvweealIit the achromatic dianad. With 

reference to the image processing modd, values of JTi desc to sere represent dark pixds mMt 

increasing positive values represent lighter and lighter gr^. 

The Ki and K» diromatic response functions shown in figure 3.2(a) and (b) respectively 

are saJlar to those derived psychophysicdly by various researdiers [18J. They are dearlj^ 

opponent .iaands and fudging by the peaks dong the wavdength azis, i, a Red-Green 

opponent d^annel and JST, i, . Blue-Ydlow opponent ch«md. Increasingly positive vdues of 
for exainple Pepre5cnt increasing levd, of Red while increasing negative vdues represent 

a««ag Jevds of Green. Ihe rdationship between the K space and perception of colour is 

snowB la figure 3.6. 
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latwtively, the perception of saturation is related to how 'Yar" a colour is from being 
gr«7. For lack of a better measure of distance, we consider a Euclidean metric to ealeoUte 
the distance from the Kl ass, giring: 



Saturation is thus related to the sum of the activities in the two chromatic channels. According 
to this metric, circles in the £s*£9 plane represent cdlonrs of equal saturation as shown in 
figure 3.6. 

To investigate this more thoroughly, we mapped these drdes to the CIE z*y plane. We 
did not obtain the same curves of constant saturation as determined psychophysically [18]. 
This seems to suggest that the Euclidean metric is not appropriate. We then mapped dlipr ss 
whose major to minor axis ratio was in the ratio of the power present in the K2 and Kz 
channels. This resulted in curves that tended towards those psychophysically determined. 
What this all means in terms of image processing is that although the Euclidean metric as 
inappropriate, increasing distance fi^m the Ki axis does represent increasing saturation. The 
fact that the metric is not Euclidean indicates that saturation is colour dependent. 

£a a like manner we define hue (E) as the ratio of the activities in the Kt and Kz channels. 
Therefore geometrically, E can be thought of as the angle B in the K^^Kz plane. 




(3.11) 
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Chapter 4 

Use of Model in Colour 
Enhancement 

4.1 Attempted Enhancement Methods 

Bcfort disraliiac tht method wt hm ntUed o& tat ohaaecmtat, we discus seant ef the 
methods thftt we attempted b«t that were waccesefid. 

4.1*1 AmpHiude Sealing 

As discQssed i& chapter St the Kt ud Kt chaa&els are oppoaent red-greca aad Uae-ydlow 
^aa&tls. Pfaysielogieellj, if we iaoease the scoral aetiTiQr 'm the EVS colour chaaads thA 
the sezksatiea of colour should isoease. Since in our model larger absolute valuu of JTs or ITs 
represent increasing amounts of colour, the sensation of colour may he increased by simply 
increasing these absolute values. In addition, because colours are represented in the EVS in 
an opponent fashion, it is possible that if we increase the iiffertnct between the opponent 
colours a richer image will result. For example in the chaancl if we mahe the reds more red 
and the greens more green, then detail revealed by colour should become more apparent. Both 
an increase of total colour and of colour diiTercnee may be aehicTed using linear amplitude 
scaling. That is, 

Xlmm*Xi^c (4.1) 

where 

S m 2,S 

m > 1 
« > 0 
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The eonstaat c represents a global xed-greea or blue-yellow colour diift and without additioaal 
knowledge about the image is usually set equal to aero. 

The effect of aquation 4.1 on colour strcngtli can be understood by substituting into 
equation 3.11 which results in 5' « mS. That is, strength is increased by a factor of m. 
AUo, since Kt and JCt are opponent channels and m > 1, awltiplicattoa by m increases the 
opponent colour difference. 

Amplitude scaling is attractire since it is a purely point-wise operation and is thus Tery 
fast to perform. The problem with it is that unlike its use in achromatic SmageSt here we do 
not know the allowable range of JTj. Therefore m cannot be calculated and must be ebtai&ed 
heuristieany.^Typieal valuu that we used were in the range of lJ2^l.i. Note that the same 
m SBust be used for and Km iu order to retain the correct colour relationshiptf. 

The overall effect of linear amplitude scaling is merely a sensation of global saturation 
or colour st^ngth increase. There is no enhancement of detatt. It may be used thou^b in 
coiuunction with the detail enhancement procedure desoibed in seetioa 4.3. 

4.1.2 Histogram Methods 

Vli also showed in chapter 3 that the frequency aaiure of the two opponent chanaels is 
lowpass. Since they are lowpass. iralues in Jfs and JTs otaxpj a limited range. Also bteanse 
of their opponent nature, values are dustcred in » small positiTe and negative region. As a 
result their histograms tend to be highly skewed with two peaks concentrated orer a small 
range as can be seen in figure 4.1. A common technique for enhancing skewed black and white 
images is to use histogram modiftcation te^qnes where the original image !s mapped so that 
the histogram of the enhanced image follows seme desired form [10]. The most commonly 
used method is histogram equilization for mhith the enhanced image histogram is forced to 
be uniform resulting in increased dynamic range. 

It seems attractive to use a similar method for colour images. Owing to the three di» 
mensional nature of colour however, naive application of histogram enhancement techniques 
will result in severe colour distortion. For example if a large number of pixels have a similar 
colour then separate equalisation on each of the colour planes will result in thue same pix- 
els now spanning a range of arbitrary unrelated eblouis. One solution to this problem hu 
been to perform a hue dependent equalisation of the saturation component of the image [15]. 
Although the K channel ruponses can be related to the sensations of hue and saturation 
(section 3.4), we have rather chosen to resolve the problem directly. We did this by designing 
three approximate histogrsm modification techniques which maintain colour integrity by tt> 
taining the original pixel ordering. In other words, if in the original image plines, the value 
of pixel I, Pi, is less than the value of pixel j then the enhanced value of pixel s, pj will be less 
than Pj. That is 
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Thus edoiipi wm bt ehi»g*d i» » eontrolled »«mer and edow xelatioiuliipi wm be in«ia. 
t«n«d. Th. thr«e teclaivi« it appnaimatt tqudixatioii. modified appwxiaute tqualisa- 
tio& aad peak shiftisc. 

ApprojdaaaU Eqii»li«»tlon 

Thi. teetaique is a aflder ibnn af •qaali«atian that n*tly alten tbe ai«iaJ hbtofria ia 
neh a way that tie reidtiag hirtogram of tie modified ehasael is clo.er to «ufonn than tbe 
origm J. Eiieatially the way thii I» doDe ii to fi»t detoBU&e tbe meaa lerel frequency of 
eee«reace fip) fcom tbe engisal iauge hiitograa. Tha all pixel* pj which an ndi that 

/(p.) > /Cp) »««• *y ^ 

i. that low count bins are increased and high eotBt bins an ndoeed towards the mean le«L 
This aethoi therelbn approximates to •qaalixatioa as a more nilbna histognm mnhs. 
Detenninatroa of the margin is a problem that is pietm dipendtBt. 

The sesuits^of this method arc that aatBration b increased » deenued in wiioas remans 
of the pietose. Thb is preferable to a f^bU satBration e ah eTtfemgnt , acUcvcd bj aqplituda 
sealing, as slightly more detail is mad* crideat. 

Modified Approximate EqnaBsaUoB 

This method is essentially the same as approximatt tqaaliiatieB except that additional bini 
we added to the lower and npper regions to extend the edoar range. This has a eoanteipart 
ia bUek and white terms to dynamie raa^e eohaacemcBt where the enhanced black and 
white image ii forced to span the range 10.255] as tmifo»ay as possible. In the can of OB 
model, the desired range of the colour dwanels is mkneim so an arbitrary a«nber of bins 
is added. The justification behind this method for colour images is that by increasing the 
colour range slightly, there would be a greater range of hues of say green thus rerealing detail 
in green regions more clearly. This docs iadnd ocenr aad does offer a slight impreremcaft on 
approximate eqaalitatien. 

Peak Shifting 

The reMoaing behind this method u similar to that of amplitude scaling. That is. we want to 
increase the opponent colour diiference. The diiferenee between peak shifting aad amplitade 
scaling is that whereas in the latter, the colour of all imagepxels U altered by the same factor 
(m), in peak shifting only the most frequently occulting pixels are changed aad then by an 
amount proportionel to their frequency of occurrence. This noa-linear method was hoped to 
better exploit the opponent colour structure. 

Peak shifting it based on the fact that since the opponent channels consist predominantly 
of two coloun. a bimodal histogram results. The peaks oa this histogram thea represent 
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the two oppeaeat cdoon. By mcreuiaf tit distuice between tbei e peaks, we in«i3e the 

oypoaeat colour difference. To do this, we fi«t identifj the two pe.k xegioni «nd holding the 

end poSatf of the»e regioai feed, we ihift the peJu apart. The aacnmt of aUft ii preparUoaal 

to the fceqneaej of oecoieaee or height. Tberefoia the higheit pdsts at the peal, are ihifted 

the fcrtheit. To eaiBre piid ordoiaj a amoothed aad pieeewiae noaotoide Teraion of the 

original histogram ii wed to ealcnlate the ahift. Thii BBOOthed hiitogram is obtained «ain< 

a regulariaatioa technique [45]. 

Aatxaaipleofthefla«oftti8ttdai«otlithoimthowafafi«ore4.1. Shifting of Jhe two 

peak Tcgioni b dearlj liaible aa wtfl m how plxd erdexiag haa baca maintained. Tht wait 

HtengTMBoflOt«fat«w<< after C->ei*waAi»ini 
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FiguTf 4.1: 
shiftizL{ 



Typical oppe&ent cbaaad Uttegram Bhowa prior to H aad after (- •) peak 



of peak shifting it not sIgaificiBtly different from amplitude sealing. 

As neither any of the previous three methods nor amplitude scaling offered satisfactory 
enhancement, a different approach had to be contidered. This is explained in the next two 
sections. 
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4.2 Spatial Chromatic Enhancement 

The eshaBcement methods briefly outlined in eeetion 4.1 ell ignore the epatiel rd&tie&ships 
between neexby pixels. Amplitude sealing is an entire!/ point-wise epcrntien; histegrem 
methods do indireetlj aiTect spatial relations but not in a waj that the eJTeet on the irisual 
system is predictable. We know howerer that spatial relationships are Tcry important to 
the EV5. We therefore consider using an enhancement technique that cmphasisu the most 
important spatial information. lb introduce the technique we Urst briefly disenss the role 
in the HVS of retinal lateral inhibition. This discussion rereali what is the most important 
spatial iafonuation and how the HVS obtuns it. We then uie this knowledge as the basis for 
our enhancement tedmtque. 

4.2.1 Th^'RoIe of Retinal Lateral Inhibitiea 

We know that spatial information is ercn mere important to the EVS than is chromatic 
information. ' Evidence for this includes how much we understand from blade and white 
images or cTm from simple line drawings. lb e]7lain the importance of spatial iaibrmation, 
Marr suggested that the Lapladan sero crossing, also known as the primal sketdht of an 
image are the basis for image understanding the EVS [46]. The aero crossings of an ' 
image are calculated by a Lapladan operator which can also be implemented by centre-, 
surround receptive iields as found in the retina. This therefore provides a rationale ibr lateral 
ishlbitien in the retina. Another rationale for lateral inhibition is provided by the fact that 
linear predictive coding used by Srinivasan under the hypothesis that the puxpose of retinal 
operations is to reduce redundancy and compress information, predicts the shape of centre* 
surround receptive fields [24]. Eekert 9t ai reconciled these two approaches by showing that 
the Lapladan and its related sero crossings are» mathematieaQy, a spedal case of predictive 
coding. In particulaTt they showed that the Lapladan is the predictive coder ibr sere noise 
images [47]. 

Thus even though it is undear whether the importance of contours in image understanding 
is a byproduct of lateral inhibition used for compresden or whether lateral inhibition exists 
solely to emphasise contours it is dear that contours carry much important information for 
the HVS. Therefore the basic spatial ezahancement method should be one that emphasises 
contours and should be performed in a way that mimiu the operation of the centre*surround 
reti&al receptive Adds. 

4.2.2 Enhancement Based on Lateral Inhibition 

As we explained in chapter 3, the modd separates a digitd colour image represented by red, 
green and blue tristimuli into three other channels, an achromatic luminance channd and two 
opponent chromatic channds. Although this is done by considering the chromatic content 
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aJone) these et&naels h%rt a defiaite tp&tial ttrueture. Bued on the isiportance of contours 
to the EV5 we propose cxpldti&g this ipatlAl ttnietiire for eahmcemcnt of the »chrom&tie 
as well «J the chromatic contest of the image. 

It is well known that In the retina wc hire honiaaact opponent ecatrt-foxTonnd recQtive 
fields. Thu enhancement based on lateral inhibitioa o& the aehrematie channel makes sose. 
Ai we disenssed in section 2.1, red*peen eentrt^nrrotmd rteeptire fields are alto found in the 
retina and there is tvidenee for the txistence of similar hlae-yeHew reeeptiTe fields, tV^wi^^Tt 
pcrhapi in luicr numbers. Th»s axmSar lateral inhibition bwed poeessinc e& t^c Axteatic 
channels shodd also c&hanee the ehromntie isfoxmation in a waj compatible with the HVS. 

Since LP C predicts eentrt-snrround rteeptire fields, the best eorapntational approaBmatien 
of the effect of lateral inhibition in the retina should be {sren hj the use of IPC. Von- 
causal two dpeosional IPC as reguirtd hj this approa^ is cox^tationaSy czpensiTc and 
tmsuitable for real-time applications [48,49]. We know however that the Laplacian is a linear 
predictor for. eero noise imafts [47]. Ifau o good approsdmation to IPC and therebj rttinal 
lateral inhihitioa can* be obtained if we nse the lapladan to generatt scro erosdnss. This 
approzxmatioB is particularly food for the ^matie channels, JTj and JSTf, beeasse as wt 
showed in section 8.8, these channels art lowpass. Eandom'noise is malnlj a hi(h frequacy 
phenomenou so Kt tnd JTs are tsscntiaSy ndselus. Ihcse noise properties are cnphaiiaed 
further in eectioa 4.6. 

In terms of image proeusing the laplacian is calculated ly conrolTing the image, Jr(<,/), 
with a digital approximation of the laplacian ppcrator. The nppreacimatien we nse ia n 
standard 3x3 mask as follows. 



X..1 



-1 f «-l 

-X -X -x 



The response of this m filter to the ideal step edge shown in figure 4.2 (a) can be seen 
in figure i.2 (b). At the position of the edge, a aero crossing response results. If we add 
this response to the step edge itself, a Maeh band type ofeiTeet simOar to that produced by 
lateral inhibition by centre-surround receptive fields is obtained. The ad^tion of the edge 
and its sere erotsing are shown in figwe 4.2 (c). The Maeh band effect can be dearly seen. 
Based on this discussion, the enhancement we propose to nse on each of the model channels 
is convolution with the lapladan operator to generate aero sossings and then addition of 
these zero crossings to the original channels. This enhancement method is illustrated in blodc 
diagram form in figure 4.3. The parameter a,- is used to control the ampL*tude of the generated 
sero crojsingf and hence the strength of the perceived Maeh bands. The numerical value of 
this parameter is discussed in section 4.4. 

An alte.-native viewpoint to considering the enhancement to be ns a result of increasing 
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Tigatt 4-2: THe sere-ereiths nspout of tbc Ltplftdu m filter to fht iiul ttn of f«l i. 

MHn nnm -na-n .*» a naro a nawa amw fir vni»?a tot win me 

fcJ a anna (M»eii) TD ra TXT nmom 
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Figure 4.3: Block diagram of the enhancement used on each of the three channels (t = 1, 2, 3) 

(i = 1, 2, 3) ,D^TUJnn THR "332 nouinn u^ja'^n nrraH^i :4.3 -ivr 



the sensation of Mach bands may be understood from the following discussion. 
A unit impulse may be expressed as, 




Any image may then be ei^ressed as, 



= high-pass + low-pass 



(4.3) 



where NA is the simple neighborhood averaging mask. Equation 4.3 shows that any image 
may be divided into a' sum of highpass and lowpass components based on 3 X 3 masks and in 
particular that the Laplacian component of an image complements the basic spatial lowpass 
component. By increasing the absolute magnitude of the Laplacian component therefore, 
we are in effect increasing the amount of high frequency power rdative to the low frequency 
power. 

Before we leave this section it is important to discuss why we have chosen to use a small 
mask. Wc use a small mask of size less than 9x9 rather than a larger mask since at the 
current level of technology it is possible to perform very fast convolution operations using 
pipelined processors only with such small masks [50, 51]. Mudi research has therefore gone 
into their use in image processing. The reason we use an even smaller 3x3 mask is that 
such a smaU mask is sensitive to very fine detail. Thus even the most insignificant edges 
are enhanced. This acute sensitivity is essential in the K2 and K3 channels. This is because 
owing to the lowpass nature of K7 and K3 edge information in these channels is very reduced. 
On the other hand, highly sensitive masks tend to increase the image noise. Again because 
of their lowpass nature, this is not a problem for K2 and Kz- This may present a problem 
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for K, for as we show in section 4.6, it contains almost all the image noise. To overcome 
this and still use 3 X 3 masks, various techniques including lowpass filtering prior to the 
Laplacian, thresholding the Laplacian generated zero crossings, an image pyr^d or even 
adaptive convolution masks [51], may be used. For standard video pictures of the kmd mos 
likely to be encountered, we have not found any of these refinements necessary for visual 
enhancement of the image. 

4.3 An Heuristic Measure of Colour Image Improvement 

The results of our enhancement method are shown in the following sections. VisuaUy the 
effects of enhancement are clear but we would also like to judge these results in a less quah- 
tative and more quantitative manner. To this aim we have investigated a measure of colour 
image improvement. Ideally this measure should be perceptually based and should be able 
to rank images according to their quality as perceived by a human viewer. That is. we would 
like some way of being able to compare an image before and after enhancement and be able 
to comment on which one is most "visually pleasing". This is an extremely difficult problem 
whose solution is not even clear for achromatic images. We have therefore had to resort to 
using a heuristic measure although as will be seen, the measure is based on human perceptual 

responses. . . 

Smce we are enhancing colour images by virtue of their colour, one measure of colour miage 
improvement is an increase in detail revealed by colour. There axe two characteristics specific 
to colour images which we consider important for revealing colour detail. The one is the 
amount of colour variance or the number of perceptually different colours present in the image. 
This hypothesis is based on the assumption that an increase in the number of perceptudly 
different colours implies an increase in the number of perceptually different objects m the 
image and therefore increased detail. The second characteristic is colour contrast by which 
we mean that the greater the "colour difference" between two objects, the better they are 
revealed. 

Colour Variance 

Recently a novel colour quantization method has been attempted [52]. In this method the 
coW image is first transformed to the almost perceptually uniform Lab space. In an ideal 
uniform space, all colours that lie within a sphere of specified radius axe percdved as eqmva- 
lent By identifying colour clusters in this space then, the image is reguantized so that only 
perceptually different colours are represented. This method is novel since it does not simply 
reduce the number of image colours to a number that is possible to display on a graphics de- 
vice (usually 256) but to that number of colours that are perceptually different. This number 
of reduced colours is what we are seeking. 
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Owing to time constraints we have not been able to implement this method however. 
Results of this method indicate though that an appreciahle increase in the number of colours 
generally also implies an increase in the number of perceptually different colours. A iirst 
approximation to this method is thus to use the total number of image colours (JV^), as the 
first component of the colour measure. 

Experience from colour quantization indicates that the effort required to transform from 
RG -B to a uniform space such as Lah or Luv is not justified as better quantization does not 
result [53]. We have therefore decided to remain in the RGB space when comting the number 
of colours. 

Before counting, we use a technique known as bit slicing to reduce the effect of noise. 
That is, we requantize the 8 bit R,G and B planes to only 4 bits by slicing off the 4 least 
significant bits. This implies a reduction of the total nimiber of possible ima^e colours from 
(2^)^ to (2^)^': That is, a variance of 15 within each image plane is considered to be as a result 
of noise only. This is in effect an initial clustering used to counteract the influence of noise 
which may be increased as a result of high frequency enhancement. The RGB colour cube 
is now divided into 4096 sub-cubes. We define the total number of colours as the number of 
these sub-cubes that are occupied by at least one pixel. The first component of the measure 
is then the fraction of occupied colour space given by, 

where -^<p<l (4.4) 
^ 4096 4096 

Average Colour Contrast 

The second component of the colour measure is that of the average colour contrast. If we have 
a coloured object on a differently coloured background then the only way to cause the object 
to stand out from the background, without changing intensity, is to increase the difference in 
colour between the two. This supposition is based on the Webcr-Fechner law which is used 
to quantify our ability to resolve the same physical stimuli that differ only in intensity. This 
law states that physical sensation is proportional to log stimulus. That is, the absolute values 
of two stimuli is not significant for detection but rather their ratio [19,33,12,46]. 

It is simplest to understand Webers law if we consider the brightness of visual stimuli. 
Consider a patch of light of brightness / + A J surrounded by a backgroimd of intensity J. 
The difference in brightness, AJ, is called the just noticeable difference (jnd) when it is the 
least amount of brightness needed to be added to J to be able to distinguish the object from 
the background. For a wide range of brightness it has been psychophysically found that 

-J- = constant 

This result is considered true for a variety of physical sensations and although it has not been 
verified for colour, it is reasonable to assume that it holds in this case too. 
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There are two perceptual qualities of colour that may be used to measure colour contrast. 
These are hue (H) and saturation (S). In most cases where an. image is to be enhanced for 
visual purposes, it is desirable not to distort the values of H since we do not wish to distort 
the actual colours. Contrast is thus mainly introduced in the saturation image. In addition, 
to measure colour enhancement alone, we wish to ignore the effects of intensity enhancement. 
We have therefore chosen to measure colour contrast on the saturation image. 

On a complex image that contains more than an object set against a backg;round we have 
to define a region of interest that can approximate to an object-badcground scenario. We 
have found that a 3 X 3 region centred on each image pixel gives good results. In each of these 
3x3 neighborhoods we calculate the Weber fraction as follows: 

^3(i,o=^E i:s{i+kj+i) (4.5) 

ASik,l) = Sik,I)-Ssik,l) (4.6) 

where 



ib = 2,3,...,Jkr-l 

/ = 2,3,...,jyr-i 

5(i, j) = saturation at point (i, j) in each 3 by 3 region 
53(4, 1) = average or background neighborhood saturation, and 

M and N are the number of pixels per row and column respectively, of the image. 
Saturation is calculated using a simple cylindrical approximation as follows, 



where intensity I is given by, 

J = 0.3.R + 0.59G + 0.11.B 
Finally, the average saturation contrast for the entire image is given as, 

One should note that there are a nimxber of different ways of defining contrast in images. 



63 

SUBSTITUTE SHEET (RULE 26) 



wo 94/11987 



PCr/US93/11146 



These include Michelsons foimtila for sinusoidal gratings, nns contrast [54], as well as various 
frequency domain methods {55]. We have chosen Webers approach as it is easy to use, gives 
good results, corresponds intuitively to the notion of contrast and also seems to be the way 
that the HVS responds to stimuli differences, at least for brightness. 



Total Measure 

Both of the components of the measure, p and T^^, are not absolute measures. That is we 
cannot comment on the quality of an arbitrary image using them. However as indicators of 
the improvement after colour enhancement we have found that an increase in the values of /> 
and does indicate an increase in perceived visual quality. The improvement indicators we 
therefore use are rather p^d and T^rei- These are given by, 

original 

and 

where cr„igirud and ^original are the measures for the original unenhanced anage and are 
given by using equation 4.4 and equation 4.8 respectively on the original image. 

Finally, to allocate a single measure of enhancement to the image we need to combine the 
two components Pra and such, that an increase in either one is reflected in the total. 
We have found that a simple multipUcation of the two components gives adequate results. 
Therefore the final contrast-density measure we use is, 

r = PrelVl'rel (^-H) 

Use of the measure is illustrated in the following sections. 

Note that p is sensitive to random noise. Thus adding random noise to a picture will 
increase p. We counter this by bit slicing and by restricting the measure to be relative. 



4.4 Enhancement of Model Predicted Channels 

In this section we test the enhancement procedure described in chapter 3 and illustrated by 
figure 3.1 on the three channels, JTi, JTj and K3, that the transform model predicts. At the 
same time we discuss guidelines for the selection of values of a,-. At the end of this section, 
we present in tabular form the values of the colour enhancement measure for each type of 
enhancement we use. 

The discussion of the different aspects of processing will mainly centre upon the "view" 
image though final results will also be shown on a number of other images. The original of the 
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"view" image is shown in figure 4.4. It is a standard 512 x 512 chron^atic image comprising 24 
bits/pixel. It was obtained using a high quaUty 3 CCD still video camera with no mtentional 
bliuxing. 




Figure 4 4: Original 512 x 512 image of the "view". Colour resolution is 24 bits/pixel. 

.tDp^-? nva-D 24 OT nW3 512 X 512 N-n rrsTJiDin ."T^-n irnim rainn :4.4 ttn 



4.4.1 Achromatic Processing 

Edge enhancement on the intensity component of the image is the standard technique used 
for colour images. It follows much of our intuitive reasoning of how to enhance edges m 
colour images as it corresponds with our daily experience of sudi pictures. For example, 
simple drawings are usually done by first sketching the outHne in pencil and then filimg m the 
patches with cdour thereby achieving a colour image with black and white edge enhancem«it. 

To our knowledge, apart from Fangeras [16,17], the only type of edge enhancement that 
has been performed on colour images is that of the intensity component. Chroma has eath« 
been ignored or even blurred by the use of subsampling as a means of image compression [56]. 
In this section we illustrate the effect of intensity edge enhancement and show why, used on 

its own, it is deficient. ^ 

Intensity edge erJnancement is done by enhancing the char_nel. When processmg the 

Ki channel, = K. and = Kz or equivalently, 02 = Q3 = 0- 

As we show in section 4.6. the value of a has a direct influence on the increase m noise 
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power after processing. For this reason, smaH values of ai are desirable. We have fomid 
that values in the range of O.S-1.2 are a good compromise between noise increase and edge 
enhancement. 

The result of enhancement on Ki only is shown in figure 4.5. A value of qi = 1.0 was 
used. Comparing figure 4.4 and figure 4.5, the effect of edge enhancement is clear. The image 




Fi£r.:re 4.5: Achromatic [Ki) enhancement of "view" image with ai = 1.0. 

.(ai = 1.0) "qirn "riu (Ki) 'onnDN -a'U :4.5 ira 



appears less blurred with far more visible detail, especially in the background. This increase 
in background detail gives the image added depth. The cost of this enhancement though 
is loss of general image colour strength. Colours seem tired and washed out. This reduced 
colour strength can be explained since it is known that as we increase intensity, saturation at 
first increases but then rapidly decreases [57;. Another disadvantage of too severe intensity 
enhancement is that hue changes of as much as 30 nm can occur as a result of the Bezdde- 
Brucke effect [57]. This means that intensity enhancement alone may not only change the 
saturation of colours but also their apparent hue. This is undesirable. 

4.4.2 Chromatic Processing 

Edge enhancement has traditionaUy not be=r. used on chromatic information for a number 
of reasons. One reason is that the existence :: colour Mach bands in the HVS has not been 
conclusively shown. Another reason is beca--. of the much r'^duced sensitivity of the HVS 
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to higher frequency chromatic contrast as can be seen in figure 2.2, For these reasons, the 
opposite of edge enhancement, image blurring, has primarily been used on the image colour 
components to achieve compression [56]. As we saw in the previous section, achromatic 
enhancement alone is deficient. In this section we show what may be achieved if we are 
prepared to enhance rather than reduce the high frequency chromatic content of chromatic 
images. 

Convolution with the Laplacian filter produces zero crossings of amplitude proportional to 
the heights of edges encountered. Since the energy of K2 is much greater than JTs, edges are 
more prominent in the K2 channel. In addition, chromatic contrast sensitivity of the red-green 
channel is known to be greater than for the yellow-blue channel (see section 2.4). Therefore 
it is expected that K2 will be dominant in increasing the perception of high frequency detail. 
The exact effect of enhancement cazmot be predicted and has to be experimentally calculated. 

Estimate of and otz 

Before testing the enhancement of chromatic information, we estimate the values of aj and 
aa. As ah-eady shown, the chromatic channels are spatially lowpass. Therefore they should 
contain little noise. High frequency enhancement should thus be able to be tolerated to a 
larger degree than in the Ki channel. Also, as can be seen from the CSF's shown in figure 2.2, 
the HVS is far less sensitive to chromatic high frequency information than it is to brightness 
information. To achieve noticeable high frequency chromatic enhancement therefore, we have 
to increase the chromatic high frequencies to a greater degree relative to the achromatic 
ones. This means that we should use larger values of aa and as relative to ai- Without 
further information we have decided to choose a? = 03. To get a feel for the range of a2 we 
compared the luminance contrast sensitivity function with the chromatic contrast sensitivity 
functions of figure 2.2. Where chromatic sensitivity starts to decline, the ratio between these 
two functions is about 2 and at high frequencies, where the curves are ahnost parallel, the 
ratio is about 6. We therefore decided to choose ai/Q2 ^ [2,6] . 

We are now in a position to perform the chromatic enhancement experiment. Since for 
the achromatic enhancement of section 4.4.1 we use ai = 1.0, here we choose = 03 = 3.0. 

Enhancement of K2 Only: 

The result of enhancement on the red-green channel only is shown in figure 4.6. If we compare 
figure 4.6 to the original of figure 4.4, there is a definite increase in image sharpness. This is 
particularly evident in areas where there is a colour change such as on the coloured towers 
and around the foreground bushes. The actual colours are very similar to the original ones. 
If we now compare this to the result of achromatic enhancement (figure 4.5) we see that edge 
crispening is not as dramatic but that colour rendering is better. The colours are less washed 
out. 
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Figure 4.6: Chromatic enhancement of the K2 channel of "view" image (02 = 3.0), 

.(a2 = 3.0) "qirn •2m {K2) ^unnj ran :4.6 ivr 

Enhancement of Only: 

The result of enhancement on the yellow-blue channel only is shown in figure 4.7. Comparing 
figure 4.7 to the original (figure 4.4), we notice that the sharpening effect of edge enhancement 
is very slight. What is dramatic is how at colour edges and around small- detail the colour 
strength is markedly increased. Thus although high frequencies do not increase much, small 
detail is made more visible by virtue of an increase in strength relative to its surround. This 
can be clearly seen in the foregroimd where in the field, rows of planes are suddenly revealed. 

Enhancement of if 2 and Kz Simultaneously: 

In figure 4.8 is the result of enhancing both of the chromatic channels simultaneously. Compared 
to the original, this image exhibits both the sharpness increase contributed by enhancing K2 
as well as the colour detail increase contributed by Kz- What is surprising though is that the 
shfiurpness of this image is much sharper than figure 4.6 or figure 4.7 alone. In fact the detail 
revealed is almost that of the achromatic enhanced image of figure 4.5. This is even more 
surprising if we remember that 'ogether K2 and K3 contain less than 1% of the total image 
energy. Yet as can be seen, thair influence on image quality is quite dramatic. Note too that 
the colour rendering of this image is much better than the achromatic only enhanced image 
(figure 4.5). 
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Figure 4.7: Chromatic eriancement of the Kz channel of "view" image (03 = 3.0). 

.(03 = 3.0) "Trn (Kz) Y"^^ 'onnj tutj :4.7 ivh 

Enhancement of All Three Channels: 

Because of the reduced sensitivity of the HVS to high frequency chromatic contrast, as dis- 
cussed in section 2.4, very high frequency image enhancement cannot be obtained by process- 
ing the chromatic channels alone. Owing to this fact and the promising result of chromatic 
edge enhancement we are led to believe that the best result would be when we combine 
achromatic and chromatic enhancement. In this way we would adiieve the additional edge 
crispening that achromatic enhancement achieves and the combination of good colour render- 
ing, small colour detail enhancement as well edge enhancement that chromatic enhancement 
offers. 

To test this we enhanced all three channels simultaneously using ai = 1.0 and a2 = 0:3 = 
3.0. The result of this combined enhancement is shown in figure 4.9. It can be seen that 
this is the sharpest image but still with good colour rendering. This image is also the most 
visually pleasing from amongs: ihe original and the three prior enhanced images (figures 4.5 
to 4.8). 

In figure 4.10 and figure 4-11 we show two more examples of the effects of chromatic edge 
enhancement. 
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Figure 4.8: Chromatic enhancement of both the K2 and Kz channels of "view" image 
(02 = a3 = 3.0). 

.(aj = Q!3 = 3.0) THu "c[irn "joj [K^y^ {Kz) 'SITU 'onra tutj :4.8 ivk 

Ranking of Results 

Here we present for easy reference tabulated results of the quality measures obtained for each 
experiment. These are shown in table 4.1. The table also contains the colour improvement 
measures from the results of the methods of section 4.1 that we attempted but that did not 
produce good results. 

From table 4.1 a number of general trends may be noted: Enhancing A% and Kz results 
in greatly increased colour contrast as well as increased number of colours while ^tiViMriTig 
Ki has little effect. Where no colour detail is improved (amplitude scaling, peak shifting, 
K\ enhance), both prei wd W^^i are low while increasing the amount of colour enhancement 
increases both of these parameters and thus also the total measure P. However the actual 
value of r is not a good indicator of visual colour image improvement since it does not reflect 
for example the marked improvement between the restxlts of the last three entries. Thus 
although r does give an indication of whether colour detail is improved, it is not a reliable 
measure of the extent of the improvement. 

Also included in the table are the results of two control experiments that we performed. 
In the first experiment we wished to check whether the same results that we obtain by edge 
enhancing K2 and coxild be obtadned by simply edge enhancing the saturation component 
of the image. This idea stems from the fact that the chromatic components K^ and Kz are 
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Figure 4.9: Combination of achromatic enhancement on the Ki channel with ai = 1.0 as 
well as chromatic enhancement on both the K2 and Kz channels of "view** image using 
02 = a3 = 3.0. 

"Hir'n mmn *?m iTs-i vzsi ^snu 'su ^mm titui Ki fra ^ 'onnDN nii) *?m aTrm :4.9 ttn 

.(ai = 1.0, Q2 s 0E3 = 3.0) 

directly related to saturation (see section 3.6). To do this, we first transformed from RGB to 
the hue, intensity and saturation space, HIS. We then edge enhanced 5 using the Laplacian 
as we did on the model channels. The result was then transformed back to RG3. Results 
were poor and did not compare well to the same procedure performed on the model chromatic 
channels. 

In the second experiment we wanted to show that colour detail is not simply enhanced by 
a mere global saturation increase. To show this, we edge enhanced the Ki channel and then 
increased global saturation by a constant multiplicative factor. Again results were poor. 

4,5 Enhancement Using a Constant transform 

In this section we show how use of the constant transform ^hat we derived in section 3.2.2 
for a set of images also resul:s in enhancement that is as good as enhancement using the 
complete model. It is important to note that the "view" image that we enhance in this 
section was not used in the derivation of the constant transform but does belong to the same 
class of natural image scenes. Therefore the enhancement is predicted to be good. 
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Figure 4.10: Chromatic and achromatic enhancement on "ship" image. Bottom rieit) original 
image. Bottom left) Result of Ki enhancement only. Top right) Result of enhancing K2 and 
Ks. Top left) Result of enhancing all three channels simultaneously. 

(Ki)'7xi Tij^j -M nNHinm .nrn^ rn'n mii?nn njinm ."n"jw"n 'onrai -onraN rcr :4,io ttk 
Ta-un ^0) nTvm "jHwrn: f JiTs)-! (JiTj) "tu 7121' nNsinn jusm rn^n ."j^nar'nan-jn Ta"?^ 



Shown in figure 4.12 is the result of enhancing the "view" image with the sa.-ne enhance- 
ment parameters, ai = 1, g; = 3 and 03 = 3 as used in the fuU transform method of 
section 4.4. 

If we compare figure 4.12 with figure 4.9 we can see that the results are almost identical. 
4.6 Enhancement in the Presence of Noise 

la this section we show how increasingly large %-aiues of a influence the image noise power 
after processing. We also illustrate how use of our method of transforming the image into an 
achromatic and two chromatic diannels results in ahnost all. the noise being in the achromatic 
channel and ahnost no noise in :he two chromatic channels. This is due to the lowpsss nature 
of the chromatic channels (see section 3.5). This leads to the conclusion that on =cisy image 
it is preferable to enhance only the chromatic channels. We demonstrate this and show that 
on very noisy images enhancen:ent on only the chromatic channels produces an image which 
appears less noisy even when using large values cf q. 
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Figure 4.11: Chromatic and achromatic enhancement on "girl" image. Bottom right) original 
image. Bottom left) Result of Ki enhancement only. Top right) Result of enhancing K2 and 
Kz' Top left) Result of enhancing all three channels simultaneously. 

' (JTi) "TV t;3^2j '^cd HNsmm rzni vn^n n^-npnn njinnn /'nnna"n ^unnni ^nmi3« tc^ij :4.ll ivh 
TUTjn "JO] nNST.n n^riD^ 'JNnmn- {Kz)-^ {K2) "n^^r •jm nnsmn ruHin ntun^ rn^n .'JNnm]: non*?n Ta^a 

.TH^ -DS^'irn nan'2m 'tv 

Before we begin, we state some well known definitions that will help in imderstanding this 
section. For a stationary random process 2(721,712), the power spectrum PJuji.u;^] is defined 
as, 

P.(wi,a;2) = -F[JEx(ni,n2)] 
and Rx{0,0) = al is called the average power of r. For a white noise process, 

f cl if Til = ^i,n2 = kz 
RAn,,n2;kuk2) = E[x{n„n,)x-{kuki)] = | otherwise 

For a stationary white noise process then, 

i^^(nx, n:: <i, = £^ , fc2)2'(fci " "i > *2 - '^2)] 
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11 operation 
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x^eaJc smlt 




1 AO 

1.08 


1 OA 

1.30 


Amplitude scaling 


1.33 


1.06 


1.40 


JCi enhance 


1.14 


l.Ui 


1.51 


Approx. equaliz. 


1.48 


1.38 


2.04 


Smear, approx. equaliz. 


1.55 


1.40 


2.17 


Sat. inc. + Inten. edge enh. 


1.93 


1.15 


2.22 


Sat. edge enh. 


1.85 


2.29 


4.23 


Kz enhance 


1.91 


4.15 


7.93 


K2 and Kz enhance 


2.95 


3.30 


9.71 


K2 enhance 


2.10 


4.69 


9.85 


Ki , K2 and enhance 


3.04 


3.31 


10.06 



Table 4.1: Values of the quality indicators in each of the enhancement ezpeziments. 

.TOTO ^'jn lUH ^33 TO^n Tra ^cd irjiD :4.l rrpno 

In other words, the power spectrum of a stationary white noise process is given by 

P.(u^i,^2) = V (4.12) 

Now if we assimie that the image noise is white stationary noise of power then it can 
be shown that the noise power of the processed image, cr^ is also white stationary noise and 
is given by, 

^p = '^EE^L- (4.13) 

« i 

where hij is the convolution mask used for processing [56,58]. 

The ezxhancement we use, as shown in figure 4.3 can be expressed by, 

K' = aK*LB + K (4.14) 

Equation 4.14 can be rewritten using the delta mask d{i^j) as, 

K' = K*id + aLs) (4.15) 

This implies that 

kij^d^oLs (4.16) 
Applying equation 4.13 to expression 4.16 and simplifying results in, 

<r2 = ^(9a^ + 2a + 8) (4.17) 
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Figure 4.12: Achromatic-chroniatic enhancement on "view" image using both the constant 
transform $c, (top right) and the full computational model (top lef: , 
.I'HTJ nTun^pn misin ^^^jn •?nnn r-r:2n?a na-u nNxinn ."qi:"n ^on"":- ^umiDN ti^^u :4.12 ti^n 
."^Hmn -:7m'7n njsin ,$e nrupn mnnnn riiusnxi 7a« na^u inw niamn 

Equation 4.17 shows that an increase in a results in a greater than squared increase in the 
processed image noise power. For noisy images therefore, a should be kept small. 

As an experiment we added to each of the -2, G and B planes of the "ship" image stationary 
white gaussian noise of variance 20. We then transformed this noisy image using the model 
to obtain the noisy channels if:-, Kzn and K^n- We estimated the nrise in each channel as 

Ni = Kin - Ki 

where Ki are the channels obtained when no noise was present. Since the transformation 
from RGB to the K space is linear, the noise in the K channels will also be stationary white 
gaussian noise [37]. The resulting noise power will then also be given by equation 4.12. 

The resulting noise variance in each of the channels was o-f ^ = 31, c;, = 0.2 and (r?^ = 0,01. 
This shows how little noise power is present in the chromatic channels compared to the achro- 
matic one. 

We then enhanced only the chromatic channels with a2 = as = 2, If we compare this 
to the noisy image we can see hew due to chromaric enhancement th^ ship appears to stand 
out from the background noise and there is no apparent baclcground n^ise increase. Thus by 
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seemingly increasing the sigr-al we have increased the signal to noise rauo without decreasing 
the noise. Apart from the low noise in the chromatic channels, another reason that high 
frequency echancement here does not degrade the image is thar the HVS is insensitiv* to 
very high frequency chromatic contrast. 

When we snhanced only zhe achromatic channel we obtained, as expected, a Texy much 
degraded image. Note that we have intentionaUy not used any noise reduction 'techniques on 
the noisy K\ channel since we only wished to illustrate the benefits of chromatic enhancement 
and not what may be achieved with further processing. 

The results of this experiment are iUustrated in figure 4.13. From this we can conclude 




Figure 4.13: Enect of chromatic and achromatic enhancement on noisv ship image. Bottom 
nght) Origmal image. Bottom left) Unenhanced noisy image. Top left) Result of chromatic 
enhancement orJy. Top right) Result of achromatic enhancement orJy. 

si^r.um n-Ts- (rcn nan*? .rr:-m "tw nmrn hdtj -tv 'onnai 'onras ^:2VJ "jca niacunn :4.i3 tm 
.'nnra r,Tv ^cc -tam C^anan nTi-n? .^nran tutj tdj nxsin (i'd-d n^un? jiujun rumn (^Knmn noD5 



that since most images do contain at least some noise, achromatic processing should be used 
to a limited exzenz and most eriiancement should be performed on the chroma. This is in 
contrast to the processing techniques currently used. We can also see that noise reduction 
techniques need ;nly be applied to the achromatic channel and not, as is commonly done, to 
each of the thre- planes. This is a great computational saving. Ar.-.:her conclusion is that 
chromatic edge ^.-Jiancement car. always be used, even on noisy images, without degrading 
the image. The izr.aze may not be enhanced but nor will it be degraded. 
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Chapter 5 

Summary and Conclusions 



Summary: 

In this report we have developed an opponent colour model of the early HVS for the purpose 
of colour image processing. We have also investigated some applications of this model to 
colour image visual enhancement, in particular, colour edge enhancement. 

The first stage of the model consists of a linear transformation of a digital image repre* 
sented by the camera or display tristimuli (RGB) to tristinouli representing the retinal cone 
responses (ViV^Vi). By assuming that the diiFerence between the CIE XYZ and modified 
XYZ tristimuli is insignificant for image processing, we were able to derive a simple linear 
transformation Tizv. As our results seem to indicate, this assumption was a reasonable one. 

The next stage of the model was' to transform from the cone responses to the opponent 
colour space. This transform is not well understood and till now all opponent colour models 
have been derived by fitting an assumed model to some psychophysical data. We on the other 
hand have based the transform on new evidence that the HVS obtains its opponent responses 
in such a way that redundant information is reduced and energy compressed. 

In terms of these requirements the Karhtmen-Loeve transform is optimal for any ensemble 
of images. Therefore we use it to derive the opponent transform matrix $ that reduces 
chromatic correlation. Transforming results in three new planes or channels labelled Ki^ 
and ifa. The KLT is performed for each image and even though it is simpler to perform than 
spatial decorrelation, the operation still takes about 1.5 minutes on a Sun*4. This generalized 
model is thus not suitable for real time use. 

We also showed however that it is possible to derive a constant transform $c that performs 
as well as the specific transform for any set of similar images. Since we wished to show at 
the same time that our model approximates to the early HVS we decided to derive $c for 
the ensemble of natural daylight images. We did this because we firstly assume that the HVS 
opponent transform is constant. This is because it is performed in real time in the retina. 
Secondly, we assume that as man has adapted optimally to his environment, the transform 
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for the natural daylight group of images should be similar to the HVS opponent transform. 
A Karhunen-Loeve analysis of such a set of images yielded $e. We showed that $e is as good 
as the specific image transform for a human viewer by showing that the same enhancement 
using both transforms yields results whose difference is imperceptible. The implication of this 
is that the generalized model can be reduced to a much simplified model involving only two 
Imear transforms represented by two 3 x 3 matrices. This model can be realized in real time. 

To show the similarity of our model to the HVS, we transformed the cone response curves 
using #c. This yielded three new spectral response curves. The response curve for the Ki 
channel was very similar to the relative luminous efficiency curve that has been psychophys- 
icaUy determined. We thus caH Ki the achromatic or Black- White channel. The spectral 
response of the channel has opponent peaks in the red-green area of the spectrum and 
is thus called the Red- Green opponent channel. For similar reasons, Kz is called the Blue- 
Yellow opponent channel. These opponent spectral response curves are also MTt^n^r to those 
psychophysically predicted. This illustrates the similarity of our model to the HVS. 

Investigating some of the properties of the model channels showed that Ki is spatially 
highpass and contains almost all the image energy. K2 and Kz are spatially lowpass and 
contain very little energy. 

As a side issue, we noted that when plotted on a log-log scale the achromatic channel 
for natural images exhibits approximately 1// behaviour as has previously been found. It is 
intersting to note though that the chrominance channels behave differently to the achromatic 
channel. Their behaviour is approximately of the form 1/2/. 

We then considered enhancing in the opponent K space and then transforming back to 
RGB space to view the results. On achromatic images, edge enhancement has been used 
for a long time with good results. This is due to the existence of Black- White opponent 
centre-surround retinal receptive fields implying luminance lateral inhibition and hence the 
importance of edge information. Until now this same method of enhancing has only been 
used on the intensity component of chromatic images. In fact colour has usually been blurred 
especially for coding purposes. Because of recent evidence of opponent colour centre-surround 
receptive fields we investigated performing colour edge enhancement too. This is done by sim- 
ulating lateral inhibition by decennining the Laplacian zero crossings of the relevant channel 
and adding these to that same channel. This has the effect of increasing perception of Mach 
bands. The Laplacian zero crossings are calculated using a small (3 x 3) digital Laplacian 
mask. Therefore the enhancement procedure is also suitable for read time operation. 

The amount of enhancement is controlled by a parameter a and we have found that for 
achromatic enhancement, ot = 1 while for chromatic enhancement a = 3 are reasonable values. 

Achromatic enhancement performed on the Ki channel produces as expected a much 
sharper image. At edges though, because of the increased intensity, colours seem faded thus 
making the whole image appear washed out. Enhancement performed on K2 alone produces 
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an image with better colours that is slightly sharper than the original. Enhancement on Kz 
produces no sharpness increase but small coloured detail is made to stand out by increasing 
its apparent colour strength. The most sxirprising result is that enhancing both chromatic 
channels simultaneously produces an image which is almost as sharp as that obtained when 
achromatic enhancement is used but with the additional property of richer colours. This is 
even more surprising when we recall that the chromatic channels contain less than 1 % of the 
total image energy. The best result is obtained when we combine achromatic and chromatic 
enhancement as then we benefit from the increased high frequency enhancement that because 
of the nature of the HVS, chromatic enhancement alone cannot provide. 

The same enhancement performed in spaces other than the HVS IC space did not produce 
comparable results and even degraded the image. This seems to indicate that without a 
quantitative defmition of subjective image quality, enhancement should be based on a visual 
model and that our visual model is a good one for image processing purposes. 

We also investigated the effect of enhancement on noisy images. We were concerned 
with this since edge enhancement generally increases high frequency noise. By adding white 
gaussian imcorrelated noise to each of the RGB planes and then transforming we estimated 
the noise in each of the K channels. This showed that Ki contains nearly all the noise and JTj 
and K3 negligibly little. This is understandable because of the highpass and lowpass nature of 
the different channels. Enhancement of the chromatic channels alone thus does not increase 
the noise much but does seem to improve the image therefore improving somehow subjective 
signal to noise ratio. An implication of this is that chromatic edge enhancement can always 
be used without fear of degrading the image. 

Finally, we investigated a quantitative measure of colour image enhancement. Presently 
the only way of evaluating results has been to use subjective responses. We wanted to use a 
quantitative measure of post processing colour image improvement. Due to the highly com- 
plex nature of human colour vision we attempted to simplify matters greatly and based this 
measure on two features that we considered to be important for colour detail improvement. 
These were the number of perceptually different colours and average colour contrast. An 
increase in both of these should indicate an increase in visible colour detail. Experimentally 
we found that this is true but that the absolute values of these parameters are not a reliable 
way of comparing one method against another. All they can tell you is whether there was 
some improvement or not but not the level of improvement. The measure is therefore not 
reliable. 

Conclusions: 

• The correspondence between the model response curves in the case of natural images 
and those psychophysically predicted demonstrate the similarity of the model opponent 
process and that found in the HVS. 
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• Chromatic edge enhancement produces results not possible by achromaric edge en- 
hancement alone. We therefore feel that when dealing with colour images it is worth 
eacploiting the additional special qualities of such images and to perform chromatic edge 
enhancement as well as the standard intensity enhancement procedures, 

• Since it is possible to simplify the opponent transform from a Karhunen-Loeve transform 
per image to a constant transform for a group of images, the model can be realized in 
real time. 

• Assimaing that a similar redundancy reduction process is performed in colour blind or 
partially colour blind people (protanopes or deuteranopes), the model can be tuned to 
enhance images for the benefit of such people. This may be done by simply redefining 
the cone response curves such that they match those of the colotxr deficient person. 

• The fact that most image noise is contained in the achromatic channel makes chromatic 
edge enhancement possible without fear of degrading the image. Thus chromatic edge 
enhancement can be incorporated as a permanent feature of image capture and display 
systems. 

• The simplicity of the enhancement procedure (3x3 mask convolution) means that it 
may also be used in real time. 

• The measure of image quality is unsatisfactory and cannot be used as a reliable estimate 
of post processing image quality. 

• Erom the amplitude spectra it seems that natural images can be typified by both the 
behaviour of the achromatic component (1//) and the chromatic components (1/2/). 

Further work: 

There are a few areas which should be investigated farther: 

• We know that the HVS not only reduces chromatic correlation but also spatial cor- 
relation. This effect should be incorporated into the opponent transform perhaps by 
performing a neighborhood Karhunen-Loeve transform rather than by performing it 
pointwise. Non-linearities of the HVS could also be incorporated. 

• We have only looked at one application of the model in colour image processing. An 
immediate further application is to the problem of quantization. Because of the way 
energy is spread amongst the three channels, simple and efficient quantization may be 
possible by allocating the most bits to the luminance channel, less to the red-green 
channel and the least to the blue- yellow channel. Other possible applications are coding 
and noise reduction. Applications of the model to the problem of colour constancy 
shotild also be investigated. 
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• The measure of enhancement is far from comprehensive and requires much more work. 
Without changing the measure, two immediate extensions would be to use a larger 
neighborhood for estimating background saturation and to weight p by the number of 
pixels of each colour so that less occupied cubes in colour space would be less significant. 
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Appendix A 

Colo ur imet ry 



As we have seen in chapter 2 the initial cone responses are only the first stage in determining 
the coloTir of an object. Notwithstanding this and the fact that colour processing in the HVS 
is poorly understood, there is a need to deal with colour in fields as diverse as television and 
paint manufacture. Therefore to standardize the naming of colours the CIE (Commission de 
L'Eclairage) adapted a standard based on the assumption of trichromacy. 

Trichromacy has been taken to imply two basic assumptions also known as Grassmans 
Laws, 

• Any colour can be matched by a vectorially additive combination of three primaries. 

• Colour processing is linear, indicating the properties of proportionality and additivity. 

The CIE standard transforms radiant energy into a three dimensional colour space. The 
equations governing this transformation are; 

L = ]j{\)Si{\)dX (A.1) 
M = f^I{X)Sm(X)dX (A.2) 
5 = JjiX)S.iX)dX ^ (A.3) 

where the scalars L.M.S are the tristimulus values and S/, Sm and 5, are three hypothesized 
colour filters which play a role similar to that of the colour absorption curves of the cone 
pigments. 

To determine 5/, and 5,, colour matching experiments using three linearly independent 
monochromatic primaries are used. The curves thus depend on the choice of primaries. In 
order to standardize and avoid negative values response values, the CIE defined the well 
known XYZ system which is based on a set of unreal primaries. As measurement techniques 
improved these 1931 CIE XYZ values were modified by Judd in 1964 to obtain the XYZ 
values. 
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A digital image is usually represented by three planes containing the tristimulus values 
related to the set of primaries used to display the image. These are usually Red, Green and 
Blue. 
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Appendix B 



Scene Independent Transform 



We wish to find an opponent colour transform that can be used for pictures of a similar type. 
To do this we propose calculating a Karhunen-Loeve transform not on a single image but on 
a set (Y) of images. To calculate the Karhunen-Loeve transform we first have to estimate the 
covariance matrix of Y. It is not feasible to simply make one super-image consisting of all 
the images in the set and then estimate its covariance matrix owing to the inmiense amount 
of data this will entail. In what follows we derive an estimate of the covariance of the set of 
images, Cy. 

In order to simplify the mathematical notation used further on we will now consider each 
X by TT image plane as a one dimensional 1 by i • W vector. Multispectral images consist of 
M planes while standard colour images have Jkf s= 3. For each image we may write 



where zj^rn is the intensity of plane m at location j in the 1 by £ • W vector. The full 
multispectral image may then be written as 



X = I X2 Xj • • • Xi,w ] 



Note that it is unimportant whether we form each vector as a row or column stack. With the 
new notation in mind we may rewrite equation 3.5 as: 



LW 



L-W i ^ 



(B.1) 
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where we similarly rewrite equation 3.6 as 



xTwFi^-^i (B.2) 



If we expand equation B.l we obtain: 

LW 

E 

and noting that 

£W \ . LW 



, LW 



1 \ t 1 



and that 



J LW 



equation B.l then simplifies to: 

Cx 



I ^ LW \ 

Now consider the set of data to consist of N images each of size X„ by W„ where 
n = 1 . . . JV. In keeping with the vector notation, each element of y can be written as: 



[ 



VnJ,7 
VnJM 



In order to simplify the development we order Y such that all the Yijs are first then the ^2^5 
and so on. That is, all the pixel vectors of each inxage are grouped together. This results in 

The total number of vectors in Y is 



d=f:L„W^ (B.4) 
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On occasion it will be simpler to index the elements of Y by a single index j where 
j = 1 . . .i. Thus it is to be understood that is the yth vector of Y. 
If we now apply equation B.3 to set Y we get 



Cy 



= 5 E E ^n..-^«.i 1 - "Wr* (B.5) 
Vn=l isl / 



but also from, equation B.3 we see tliat 



E ^"J^nj = •CnW'n (Cn + m^m/) (B.6} 



Li addition, if we apply equation B.2 to Hut set of data we get 

= 3 ^".^ 



= 4E"^ (B.7) 

where rrin is the mean of each image in the set. That is, the mean of the set of images is the 
mean of the means of each image. 

Now substituting equations B.6 and B.7 into equation B.5 

C„ = 1 ^ X,W„ (Cn + m„W) - j5 (f: m„) nj^ (B.8) 

If we restrict ourselves to images of equal size that is, LiW^ = L2W2 = . • .LpfW/^ = LW 
then £rom B.4, 

d=NLW 

and equation B.5 simplifies to 

= ^ [E (Cn + mnW) - J (E »«n) m„*) j (B.9) 
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Chromatic Bnliancement Processor 
Hardware Description 

The CE processor is described in Figiore 1. It comprises a color converter 12, a 
decorrelating transform calculator 80, a 

decorrelating transformer 82, enhancement processor 40, Inverse decorrelating 
transformer 50, and inverse color converter 60. 

COLOR CONVERnER 12 

The color converter 12 converts the acquired color signal, typically RGB or YCA» 
to a representation similar to that of biological retinal receptors. One possible 
embodiment of this converter comprises a 3x3 matrix multiplier, converting the 
three color components (e.g. R, G, B) at each image point to three new 
components V„ V„, V^, according to a linear transform. The rationale for such 
embodiment as well as the derivation of the transform values, are detailed in 
Stuart Wolf s thesis, "Development of an Adaptive Opponent Color Model for Color 
Image Processing," presented to the Department of Electrical Engineering, 
Technion-Israel Institute of Technology, Haifa, Israel in June 1992 IW92]. The 
transform values are given below: 
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T = 



2 .5653e-l 
4 .2417©-! 
3 .6682e-8 



3.8702e-l 
2 .5035e-l 
2.45826-4 



7 .9427 6-1 
1.77086-1 
4.15046-3. 



The inverse color conversion can also comprise a 3x3 matrix multiplier, with 
values of the inverse matrix of T. It can be seen above that T is non-singular and 
thus dlvrays has an inverse. 

Alternative embodiments of the color converter 12 can take into accoimt spatially 
distributed information, and thus compute the three color components of any 
image pixel based not only on its RG.B values but also on the RG.B values of its 
neighbors, as well as on color statistics of the whole image. 

DECORRELATTNG TRANSFORM CALCULATOR 80 
The decorrelating transform reduces to a minimum the 

cross-correlation between the various color components and/or the spatially close 
pixel elements of the image signal. In an example embodiment we employ the 
Karhunen-Loeve Transform (KLT) to decorrelate the color components of the 
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image signal. Given that the color of each pixel can be represented by three 
elements, the KLT is a linear transform in the form of a 3x3 matrix ^ which 
multiplies the tricolor vector. The Transform Calculator 80 computes the values 
of the 3x3 transform matrix 

In another example, a similar technique is used to decorrelate also spatial 
correlations of the colors of neighboring pixels. 

The calctilation comprises four steps: 

(a) Computation of the mean color vector M; 

(b) Computation of the autocovariance matrix C: 

(c) Computation and normalization of the eigenvalues of C, which 
constitute the rows of O; 

(d) Computation of the inverse O'^ (in [W92] it is shown that the inverse 
always exist). 

(al Computation of the mean color vector M 

The ensemble of color vectors X consists of all the color triplets (e.g. Vi,V„,VJ over 
all image pixels. To simplify nomenclature, let: 
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X = 



Thus, there are LxW triplets for an LxW image. The mean color vector is: 



LxW ^ 



pixsXs 



This is computed by summing each color component over the whole Image and 
diving the sxmi by the number of pixels. It is well known how to devise hardware 
to perform this computation. 



fb) Computation of the autocovariance matrix C 



The autocovariance matrix C is defined as follows: 
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C = £■[ (X-M) (X-M) "^l = E{XX -MM ' 



Substituting: 





a" 


[a jb c] 


aJb ac 


XX ^ = 






ai? Jb2 jbc 




c 




ac be c*. 



we get: 
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C = 



LxW ^ 



All 

pixels 



= (— ) r 



all 
image 
pixels 



ah ac 
ah 2?2 he 
ac he 



d e f 
g h i 

b' i. 



fc) Computation and normalization of the eigenvalu es of C 



It is well known that in order to find the eigenvalues of C we must solve for all 
roots X of 



determinant ( C - XI ) = 0 



where I is the 3x3 unit matrix. In other words, solve for all three roots of the cubic 
equation: 

+ (d+A+i) (^e+fj+iJt-dh-di-AI) X+ {dhl-^eij-^fgk-fhj'egl-dik) =0 
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Since C Is real and symmetric (namely C=C*). all three solutions are real. Two 
alternative computation techniques exist for the solutions of cubic equations: 

Solution 1 : 

Given the cubic equation in simplified form. 
+ aX* + bX + c = 0 

let 



9 

9ab-27 c-2a^ 
^" 54 

then the solutions are: 
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The implementation of this computation is straightforward, and can 
be devised by interconnecting adders, subtracters, multipliers, 
dividers, square root and cube root computing elements, and lookup 
tables, according to the formulae above. 

Solution 2 : 

Given that all our roots are real and unequal, then Q^+R^ < 0. Let 



This computation can also be devised as an interconnection of 
adders, subtractors, multipliers, dividers, square root computing 
elements and possibly lookup tables for the cosine function. 



cose= 



The three roots can also be computed as: 



X2«2v^os (-|B+120'*) 
X3=2/=Scos(^B+240**) 
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Having foirnd the eigenvalues ^i. X^. and X3, we now compute the three 
eigenvectors as foUows: For each of the roots K (n=l,2,3), we compute 



'a2 



'n3l 



fih-X^) -ei 
eg-[d-kj (ii-V 



This computation can be devised by simple interconnection of adders, 
subtracters, multipliers and dividers. 

As the last step, the eigenvectors are normalized by dividing each element by the 
norm of the corresponding vector: 



The rows of the transform matrix * are these eigenvectors, ordered such that the 
corresponding eigenvalues X„ are in descending order. 
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fdl Computation of the inverse 3)'^ 

The computation of the inverse of the 3x3 matrix <I> is well known and can be 
performed by an interconnection of adders, subtracters, multipliers, and dividers. 



DECORRELATING TRANSFORMER 82 

One embedding of the decorrelating transformer is a 3x3 matrix mialtipUer, which 
converts the three color components of each pixel to another set of three 
components, decorrelated. This is especially applicable in the case that the 
decorrelating transform is the KLT described above. 

It will be appreciated that the KLT, when applied to many types of images in the 
manner described above, produces three components which strongly relate to the 
achromatic image contents, to the red-green chromatic data, and to the blue- 
yellow chromatic data, respectively, in a manner reminiscent of the opponent 
color mechanism of the human vision system, as described for example by David 
H. Hubel in "Eye. Brain and Vision". Scientific American Library, 1988. Thus, the 
three components are also referred to as one achromatic "channel" and two 
chromatic ones. 
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Another embodiment takes into account a complete neighborhood around a pixel, 
and produces a converted triplet of chromatic values for the pixel. 

Other embodiments perform non-linear transfonns. namely, more complex, 
transforms v^ch cannot be achieved by simple matrix multiplications. 

ENHANCEMENT PROCESSOR 40 

The enhancement processor can apply any desirable enhancement to each of the 
three color components of each pixel. Alternative methods for enhancements exist 

One example is the chromatic enhancement as described in [W92J. It applies 
high-pass spatial filtering to each of the components. As a resiolt details are 
easier to observe and colors are turned more vivid. As is widely known, high-pass 
spatial filtering of two-dimensional signals can be achieved in a number of ways, 
including linear convolutions. 

Another example, also explained in IW921, is chromatic enhancement in the 
presence of strong noise. In such a case, since empirically it was found that most 
noise data is confined to the achromatic channel, only the two chromatic 
channels are enhanced, while the achromatic channel is either low-pass filtered 
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or is left as is. 

In a third example, each of the chromatic chamiels would imdergo a different 
transform, depending not only on Itself but also on the other two channels, in 
order to change the color of objects, so as to make the image more readily visible 
to people having certain degrees of "color blindness". For instance, gradients on 
the red-green channel would be detected and transferred to the other two 
channels for people with red-green color blindness. 

INVERSE DECQRRELATING TRANSFORMER 50 & INVERSE COLOR 
CONVERTER 60 

Following enhancement in the transform domain, the image is transformed back 
to the original color coordinate. This is done by applying the Inverse of two 
transforms {^'^ and T^]. 



103 



SUBSTITUTE SHEET (RULE 26) 



wo 94/11987 



CLAIMS 



PCT/US93/11146 



1. Color image enhancement apparatus comprising: 
apparatus for receiving signals representing a color^ 

image; 

image processing apparatus, employing the received' 
signals, for image processing of the high spatial frequency 
chromatic components of the color image; and 

apparatus for providing a color image using the output 
of said image processing apparatus, 

2. Apparatus according to claim 1 wherein said apparatus 
for providing a color image also employs an achromatic component 
of the color image which was not processed by said image 
processing apparatus. 

3. Apparatus according to claim 1 or claim 2 wherein said 
apparatus for providing a color image also employs an achromatic 
component of the color image which was processed by said image 
processing apparatus. 

4. Apparatus according to any of the preceding claims and 
wherein said apparatus for providing a color image also employs 
an achromatic component of the color image. 

5. Apparatus according to any of claims 1-4 wherein said 
apparatus for receiving comprises: 

apparatus for receiving signals representing a color 

104 



SUBSTilUTE SHEET (RULE 26) 



wo 94/1 1987 PCr/US93/l 1 146 

Image in a first color space; and 

apparatus for transforming the received signals from 
the first color space into a color space which simulates the 
cones of the human visual system. 

6. Apparatus according to claim 5 wherein the first color 
space comprises an RGB space. 

7. Apparatus according to claim 5 or claim 6 wherein the 
cone-simulating color space comprises a (Vj^, "Vj^, Vg) color space. 

B. Apparatus according to any of claims 5-7 and wherein 

said apparatus for providing comprises apparatus for transforming 
the output of the image processing apparatus from the cone- 
simulating color space to a second color space. 

9. Apparatus according to claim 8 wherein said second 
color space comprises an RGB space. 

10. Apparatus according to any of claims 1 -9 wherein the 
apparatus for receiving comprises apparatus for at least 
partially decorrelating signals representing the color image. 

11. Apparatus according to claim 10 wherein said apparatus 
for decorrelating comprises apparatus for performing a Karhunen- 
Loeve transform. 



105 

SUBSTITUTE SHEET (RULE 26) 



wo 94/1 1987 PCr/US93/l 1 146 

12. Apparatus according to claim 11 wherein the apparatus 
for providing comprises apparatus for performing an inverse of 
the Karhunen-Loeve transform* 

13. Color image enhancement apparatus comprising: 
apparatus for image processing of the high spatial 

frequency chromatic components of a color image. 

14. Color image enhancement apparatus comprising: 
apparatus for image processing of the high spatial 

frequency chromatic components of a color image, whereby a 
r&oi.rlting enhanced color image is produced at least mainly from 
said high spatial frequency components. 

15. Color image enhancement apparatus according to any of 
the preceding claims and wherein said high spatial frequency 
components comprise the top half of the spatial frequency range 
of the color image. 

16. Color image enhancement apparatus according to any of 
the preceding claims 1-14 and wherein said high spatial 
frequency components comprise the top third of the spatial 
frequency range of the color image. 

17. Color image enhancement apparatus according to any of 

♦ 

the preceding claims 1 - 14 and wherein said high spatial 
frequency components comprise the top quarter of the spatial 
frequency range of the color image. 
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18, Color image enhancement apparatus according to any of 
the preceding claims and wherein the power spectrum of said high 
spatial frequency chromatic components is similar to that of a 
high spatial frequency achromatic component of said color image, 

19, Color image enhancement apparatus for modifying a color 
image for perception by a color-blind individual, the apparatus 
comprising: 

apparatus for receiving signals representing the color 

image; 

image processing apparatus comprising apparatus, 
employing the received signals, for modifying the color image 
such that at least one color in the color image which a color- 
blind individual cannot differentiate is transformed to at least 
one color which the color-blind individual can differentiate; and 

apparatus for providing a color image using the output 
of said image processing apparatus. 

20. Apparatus according to claim 19 and also comprising 
apparatus for identifying at least one color which the color- 
blind individual does not differentiate. 

21. Apparatus according to claim 19 or claim 20 wherein 
said apparatus for modifying comprises apparatus, employing the 
received signals, for modifying the color image such that a color 
in the color image which a dichromate does not differentiate is 
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transformed to at least one color which the dichroraate can 
differentiate . 



22. Apparatus according to any of claims 19 - 21 wherein 
said apparatus for modifying comprises apparatus, employing the 
received signals, for modifying the color image such that at 
least one color in the color image which a monochromate does not 
differentiate is transformed to a color which the monochromate 
can differentiate. 

23. Apparatus according to any of claims 1-22 wherein 
said image processing apparatus comprises apparatus for enhancing 
chromatic differences between spatially adjacent colors. 

24. Apparatus according to claim 23 wherein said apparatus 
for enhancing chromatic differences comprises apparatus for 
enhancing chromatic differences between spatially adjacent colors 
which are achromatically indif f erentiable. 

25. Apparatus according to claim 23 or claim 24 wherein 
said apparatus for enhancing chromatic differences comprises 
apparatus for enhancing chromatic differences between spatially 
adjacent colors which are achromatically dif f erentlable. 

26. Apparatus according to any of claims 23 - 25 and also 
comprising apparatus for enhancing primarily a high spatial 
frequency portion of an achromatic component of the color image. 
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27. Apparatus according to any of claims 23 - 25 and also 
comprising apparatus for enhancing in the presence of noise 
comprising apparatus for enhancing primarily a low spatial 
frequency portion of an achromatic component of the color image. 

28. A method for color image enhancement comprising the 
steps of: 

receiving signals representing a color image; 

employing the received signals for image processing of 
the high spatial frequency chromatic components of the color 
image ; and 

providing a color image using the output of said 
apparatus for image processing. 

29. A method according to claim 28 wherein said step of 
providing a color image also employs an achromatic component of 
the color image which was not processed in said image processing 
step. 

30. A method according to claim 28 or claim 29 wherein said 
step of providing a color image also employs an achromatic 
component of the color image which was processed in said image 
processing step. 

31. A method according to any of the preceding claims 28 - 
30 and wherein said step of providing a color image also employs 
an achromatic component of the color image. 
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32. A method according to any of claims 28 - 31 wherein 

said step of receiving comprises the steps of: 

receiving signals representing the color image in a 
first color space; and 

transforming the received signals from the first color ^ 
space into a color space which simulates the cones of the human 
visual system. 



33. A method according to claim 32 wherein the first color 
space comprises an RGB space. 

34. A method according to claim 32 or claim 33 wherein the 
cone*simulatlng color space comprises a (V^, V^^^ Vg) color space. 

35. A method according to any of claims 32 - 34 and 
wherein said step of providing comprises the step of transforming 
the output of the employing step from the cone-simulating color 
space to a second color space. 

36. A method according to claim 35 wherein said second 
color space comprises an RGB space. 

37. A method according to any of claims 28-36 wherein the 
step of receiving comprises the step of at least partially 
decorrelating signals representing the color image. 

38. A method according to claim 37 wherein said step of 
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decorrelating comprises the step of performing a Karhunen-Loeve 
transform. 

39. A method according to claim 38 wherein the step of 
providing comprises the step of performing an inverse of the 
Karhunen-Loe ve transform • 

40. A color image enhancement method comprising the step 
of: 

image processing of the high spatial frequency 
chromatic components of a color image. 

41. A color image enhancement method comprising the step 
of: 

image processing of the high spatial frequency 
chromatic components of a color image, 

whereby a resulting enhanced color image is produced at 
least mainly from said high spatial frequency components. 

42. A color image enhancement method according to any of 
the preceding claims 28-41 and wherein said high spatial 
frequency components comprise the top half of the spatial 
frequency range of the color image. 

43. A color image enhancement method according to any of 
the preceding claims 28-41 and wherein said high spatial 
frequency components comprise the top third of the spatial 
frequency range of the color image. 
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44. A color image enhancement method according to 
the preceding claims 28 - 41 and wherein said high 
frequency components comprise the top quarter of the 
frequency range of the color image. 

45. A color image enhancement method according to any of 
the preceding claims 28 - 44 and wherein the power spectrum of 
said high spatial frequency chromatic components is similar to 
that of a high spatial frequency achromatic component of said 
color image. 

46. A color image enhancement method for modifying a color 
image for perception by a color-blind individual, the method 
comprising the steps of: 

receiving signals representing the color image; 

image processing the color image, comprising the step 
of employing the received signals for modifying the color image 
such that at least one color in the color image which a color- 
blind individual cannot differentiate is transformed to at least 
one color which a color-blind individual can differentiate; and 

providing a color image using the output of said 
apparatus for image processing. 

47. A method according to claim 46 and also comprising the 
step of identifying at least one color which the color-blind 
individual does not differentiate. 
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48. A method according to claim 46 or claim 47 wherein said 
step of employing comprises the step of employing the received 
signals for modifying the color image such that a color in the 
color image which a dichromate does not differentiate are 
transformed to at least one color which the dichromate can 
differentiate. 

49. A method according to any of claims 46 - 48 wherein 
said step of employing comprises the step of employing the 
received signals for modifying the color image sjach that at least 
one color in the color image which a monochroraate does not 
differentiate are transformed to a color which the monochromate 
can differentiate. 

50. A method according to any of claims 28 - 49 wherein 
said step of image processing comprises the step of enhancing 
chromatic differences between spatially adjacent -tcolors. 

51. A method according to claim 50 wherein said step of 
enhancing chromatic differences comprises the step of enhancing 
chromatic differences between spatially adjacent colors which are 
achromatically indiff erentiable* 

52. A method according to claim SO or claim 51 wherein said 
step of enhancing chromatic differences comprises the step of 
enhancing chromatic differences between spatially adjacent colors 
which are achromatically dif f erentiable. 
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53. A method according to any of claims 50 - 52 and also 
comprising the step of enhancing primarily a high spatial 
frequency portion of an achromatic component of the color image. . 

54. A method according to any of claims 50 - 52 and also 
comprising a step of enhancing in the presence of noise 
comprising the step of enhancing primarily a low spatial 
frequency portion of an achromatic component of the color image. 
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